Skip to content

identity: a differing identifier is a contradiction, never a tolerated gap - #1743

Merged
abrichr merged 7 commits into
mainfrom
claude/identity-identifier-contradiction
Nov 4, 2026
Merged

abrichr merged 7 commits into
mainfrom
claude/identity-identifier-contradiction

Conversation

@abrichr

@abrichr abrichr commented Nov 3, 2026 •

Copy link
Copy Markdown
Member

Wrong-patient fix, found by running OpenAdapt on a real Windows back-office app in the lab.

The hole.

  • A row band padded with static label text, recorded Smith John 15 47 Hint: enter values in multiple boxes., verified against live Smith John 16 50 Hint: enter values in multiple boxes.. That is a same-name record with a different number and age, and it verified on both the plaintext and the hashed-template paths.
  • The unmatched 15 47 counted as a 4-character uncovered run, within UNCOVERED_RUN_CAP. Its 2-character tokens were below every contradiction rule.

Change.

  • identity._contradicted and identity_template._contradicted_template: an unmatched recorded token that carries a digit (id, date, age, count), of any length, is contradicted when the live band shows an unexplained digit-bearing token.
  • band_match / _band_match_template: an unmatched digit-bearing recorded token with nothing in its place sets identity_evidence_short, so the OCR tier abstains. It is never a tolerated uncovered run.
  • repair/self_repair._uncovered_band_matches: the word a dialog cuts (1984-) is already checked as a prefix of the recorded word, so the template now weighs whole words only.

Tests. tests/test_identity_identifier_contradiction.py (20 tests; 12 fail on main). Cases:

  • the found pair;
  • label padding before, after and on both sides;
  • a glued label;
  • PatNum only, age only, DOB, MRN, account, single digit;
  • 1/l and 0/O confusions on identifiers;
  • an id replaced at another position;
  • absent ids;
  • true-row controls.

Validation (details in the private evidence repo):

  • The identity evaluation found no false accepts. Stricter digit checks also refuse some same-record pairs that previously passed; this is an intended fail-closed trade-off.
  • Every committed VERIFIED OCR identity check that could be re-run with this change still verifies.

Draft until the full identity suite and an independent adversarial review are in. One open design question, whether static label text should count toward coverage at all, is separate from this fix.

…d gap

Found on Open Dental in the Windows lab: a row band padded with a static
label let a same-name sibling with another PatNum and age verify. "15 47"
vs "16 50" was a 4-char uncovered run, within UNCOVERED_RUN_CAP, and
2-char tokens were below every contradiction rule.

- An unmatched recorded token carrying a digit (id, date, age, count), of
  any length, is contradicted when the live band shows an unexplained
  digit-bearing token: a differing value is a different record.
- Absent outright (OCR dropout), it marks identity evidence short, so the
  OCR tier abstains; it is never a tolerated uncovered run.
- Same rules in the hashed-template path.
- The covered-band repair weighs whole words only: the word the dialog
  cuts is already checked as a prefix of the recorded word.
…; params keep short ids

Review blockers on this PR:
- a second copy of the recorded value elsewhere in the band ("15 patients
  found") explained the recorded id, so a sibling's differing id went
  uncharged; a digit-bearing recorded token now matches one observed copy of
  its own, on both paths;
- param mode dropped a short id lying inside the example value ("15" in
  "20150101"); a digit token is never dropped for being inside the value.
Should-fix: split and join never glue two digit-bearing tokens ("15" "47"
is not id 1547).
@abrichr

abrichr commented Nov 3, 2026

Copy link
Copy Markdown
Member Author

Independent adversarial review, round 1: blockers. Fixed in 0879718.

  1. A second copy of the recorded value explained the recorded id. If the same value also appeared elsewhere in the band (a count, a pager, another column), a sibling's differing id went uncharged. Now a digit-bearing recorded token matches only one observed copy of its own, on both paths.
  2. Param mode dropped a short id that lay inside the example value (the substring test). A digit token is never dropped for being inside the value now.

Should-fix also done: split and join never glue two digit-bearing tokens, so 15 + 47 no longer reads as 1547, and the reverse.

Each case is a regression test on both paths. A re-review follows once the full identity suite and the validation re-run finish.

Noted, not changed:

  • Swapped values within the band match regardless of order. That is a separate position-aware fix, reported to the coordinator.
  • False-stop costs: a counter that legitimately changes in the band ("3 unread" → "4 unread"), an age that rolls over, or a changing last-visit date now refuses where it used to verify. That is fail-closed.
  • The self_repair change was checked with no hole found.

One id split by OCR ("100" "512" vs "100512") and two values beside a
record whose id is 1547 ("15" "47") read alike; such a split or join now
matches but flags the fragments glyph-ambiguous, so the band abstains, as the
9th-reopening split tests require, instead of verifying.
…self

Review blocker: the similarity fallback still dropped a PatNum near the
example account number (547 vs 15470), so a sibling's differing PatNum
contradicted nothing. A digit token now belongs to a parameter only when the
value is inside it or it is the value read with OCR confusions (l5470).
@abrichr

abrichr commented Nov 4, 2026

Copy link
Copy Markdown
Member Author

Independent adversarial review: Request changes

I diffed head 25e6f8c53 against origin/main and probed the plaintext path (verify_target_identity) and the hashed path (build_identity_template + verify_template_identity) against main with targeted pairs and a short randomized fuzz (wrong record = one field changed, plus OCR noise, glue/split, lowercase, label padding and extra trailing text).

Fixes from earlier rounds hold. Each of these refuses or abstains on both paths, and main verified each one:

  • Smith John 15 47 15 patients found → Smith John 16 47 15 patients found
  • Smith John 15 47 Active → Smith John 1547 Active (glued fragments abstain)
  • param dob 2015-01-01→2016-01-01 over Smith John 15 47 2015-0l-01 Hint
  • param acct 15470→99999 over Price Mark 547 l5470

In the fuzz, no wrong record verified here that main refused. The change is stricter than main on every path I tried.

Blockers

1. A copy of the recorded value elsewhere in the live band still stands in for the sibling's differing value. This happens when the recorded band did not include that copy. The one-copy rule only helps when the recorded band holds both copies. Every recorded digit token then matches, so the sibling's unexplained id is never charged. Both paths return verified; main does too. This is the class the new code comment says is closed.

recorded live (different record) plaintext / hashed
Smith John 15 47 Hint: enter values in multiple boxes. Smith John 16 47 Hint: enter values in multiple boxes. 15 patients found verified / verified
Smith John 03/04/1980 Active Smith John 03/05/1980 Active 03/04/1980 verified / verified
Smith John 15 Active Smith John 16 15 Active verified / verified

Suggested fix: when every recorded digit token matched but the live band has an unexplained digit-bearing token, set identity_evidence_short, so the band abstains and never verifies. At minimum, do this when the unexplained token has the shape of a matched recorded digit token (same length, or both date-shaped). Make the same change in _band_match_template.

2. The one-copy assignment takes the first canonical-equal live token, not the raw-equal one, so an identical re-read now refuses. The new break in the single-token pass gives a digit-bearing recorded token the first live token that is canonical-equal to it. That token can be an alphabetic one (Bo ≡ 80, Li ≡ 11) or a look-alike. The raw-equal copy is never reached, the token is marked suspect, and the band returns mismatch:

recorded live (same record) main this PR (plaintext / hashed)
Price Bo 80 03/04/1980 identical verified mismatch / mismatch
Smith John 25 47 Active Smith John 2S 25 47 Active verified mismatch / mismatch
Lee Li 11 03/04/1980 Active identical abstain mismatch / mismatch

In the Li 11 case, the second 11 is left unexplained and is also read as a generational suffix (II). Suggested fix: assign raw-equal copies in a first pass, then canonical-equal copies, in both _match_tokens and _match_tokens_template. Add the identical-band cases above as tests.

Should fix

  • Glued id and initial now returns mismatch. Hint: enter values in multiple boxes. Nguyen Ivan 9 E → ... Nguyen Ivan 9E returns mismatch on both paths (main verified). The 2-char concat 9e is under MIN_BLOCK, so the split pass skips it, and the new digit rule then reads 9E as a differing value. Abstain is the right result for a glued read. Let the split pass try short concats that carry a digit, or downgrade to abstain when the unexplained live token starts or ends with the unmatched recorded digit token.
  • The param fallback now keeps demo-value fragments. The non-verbatim _token_belongs_to path keeps any digit fragment of the demonstrated value that is not the whole value. For example, 01 is kept when OCR split the demonstrated 2015-0l- 01. The true record then refuses (main abstained). This fails closed, so it is low priority.
  • Swapped digit values verify, as on main. Smith John 15 16 ... → Smith John 16 15 ... verifies because matching ignores order. That is a design limit, but worth one line in the docs or a test that pins the behaviour.
  • A 3-character first name replaced by a 2-character token verifies on both paths, as on main. Okafor Ann 05/14/1988 Active → Okafor Bo 05/14/1988 Active. The absent-name cap tolerates Ann. Bo is too short to be name-shaped and is a different length, so the short-token replacement rule misses it. This is not introduced here; a follow-up is fine.
  • self_repair._uncovered_band_matches now drops a cut name word as well as a cut date or id. For example, Joh for a recorded Johnathan was an unexplained name token before. This only routes the halt, and the probe re-runs the full identity check, so it is acceptable. The docstring should say it applies to any cut word.

…abstains; a 2-letter name replaces a longer one
@abrichr

abrichr commented Nov 4, 2026

Copy link
Copy Markdown
Member Author

Independent adversarial review (head d9987a8b2): Request changes

I diffed against origin/main and probed the plaintext path (verify_target_identity) and the hashed path (build_identity_template + verify_template_identity) against main, with targeted pairs and a short randomized fuzz (one field changed, OCR confusions, lowercase reads, extra trailing text).

The three fixes from the last round

All three hold on both paths:

fix recorded → live main this PR (plaintext / hashed)
1. extra copy of the recorded value Smith John 15 47 Hint: enter values in multiple boxes. → ... 16 47 ... 15 patients found verified abstain / abstain
1. Smith John 03/04/1980 Active → Smith John 03/05/1980 Active 03/04/1980 verified abstain / abstain
1. Smith John 15 Active → Smith John 16 15 Active verified abstain / abstain
2. exact copy first (same record) Price Bo 80 03/04/1980 → identical verified verified / verified
2. Smith John 25 47 Active → Smith John 2S 25 47 Active verified abstain / abstain
2. Lee Li 11 03/04/1980 Active → identical abstain abstain / abstain
3. 3-letter name → 2-letter Okafor Ann 05/14/1988 Active → Okafor Bo 05/14/1988 Active verified mismatch / mismatch

The plaintext and hashed paths agreed on every pair and every fuzz case. Nothing crashed, including empty and punctuation-only bands. Nothing verified here that main refused.

Blockers

These are left open rather than introduced: main verifies each one too. Each is in a class this PR says it closes, so the claim does not hold yet.

1. A recorded word that OCR-folds to the record's differing id explains that id, so the extra-digit rule never fires. In the single-token pass, a recorded alphabetic token still marks every canonical-equal live token as explained, including digit tokens (Bo ≡ 80, Io ≡ 10, So ≡ 50). With a copy of the recorded id elsewhere in the live band, every recorded token matches and no live digit token is left unexplained.

recorded live (different record) plaintext / hashed
Price Bo 81 Active Price Bo 80 Active 81 found verified / verified
Price Bo 81 03/04/1980 Active Price Bo 80 03/04/1980 Active 81 verified / verified
Smith Io 15 Active Smith Io 10 Active 15 verified / verified

Suggested fix: a live digit-bearing token counts as explained only by a digit-bearing recorded token (or by a split or join that already abstains), in _match_tokens, _match_tokens_template and both _extra_digit_token checks.

2. The name-replacement rule still misses a 1-letter replacement and the reverse direction (a 1–2 letter recorded name replaced by a longer token that is not name-shaped). Fix 3 counts unexplained 2-letter live tokens for recorded names of 3 or more letters only. A recorded name of fewer than 3 letters goes through the same-length short-token count, and a lowercase live token is not name-shaped. So the replacement is charged only by the absent-name cap, which tolerates it.

recorded live (different record) plaintext / hashed
Okafor Ann 05/14/1988 Active Okafor J 05/14/1988 Active verified / verified
Price Al 02/16/1963 76 Active Price J. 02/16/1963 76 Active verified / verified
nguyen al 74 03/04/1980 pending nguyen joan 74 03/04/1980 pending verified / verified

Okafor A. 05/14/1988 Active already returns mismatch here while Okafor J ... verifies, so the boundary is arbitrary. Suggested fix: while an alphabetic, name-plausible recorded token is unmatched and any unexplained alphabetic live token exists (any length, any case), set identity_evidence_short, so the band abstains. Keep the existing mismatch rules as they are, and make the same change in _band_match_template.

Should fix (fail closed, not blocking)

  • An OCR split of a 2-digit token now returns mismatch where main verified. For example, Smith John 15 47 → Smith John 1 5 47 and Smith John 15 47 → Smith John 15 4 7. The join pass skips concatenations shorter than MIN_BLOCK, so the fragments stay unexplained digit tokens and the new digit rule calls them a different value. Let the join pass try short digit concatenations; the glue rule then makes them abstain.
  • A truncated read of a 3-letter name now returns mismatch where main verified. For example, Okafor Ann ... → Okafor An .... Abstain is the better result when the unexplained 2-letter token is a prefix of the unmatched recorded token.
  • A replacement name explained by a copy of a recorded word still verifies. For example, Okafor Ann 05/14/1988 OK → Okafor OK 05/14/1988 OK. The rule for alphabetic tokens marks every copy, so the replacement is explained. This is contrived and low priority.
  • False-stop costs to note in the PR body: any extra digit token in the live band now abstains, for example a clock (... Active 10:42) or a pager (Page 2).

abrichr added a commit that referenced this pull request Nov 4, 2026
70 runs per build on the Windows 11 x64 laptop, aardwolf from the Mac,
rdp_avc_real's protocol: 40 saves, 20 false stops (decoy 10, 1440x900 10),
10 stops on the look-alike, 0 wrong records, on both builds, with identical
stop messages. The false stops come from a postcondition on an auto-selected
row and a refused session size, not from identity checks.
abrichr added a commit that referenced this pull request Nov 4, 2026
… RDP) (#1798)

Four OpenEnv environments on two Windows line-of-business apps, under the same task, `verify.yaml` and reward contract as the browser environments. The reward is verify-run's verdict on the app's own database, read outside the agent's session.

| Environment | App | The agent observes |
|---|---|---|
| `win-patient-chart` | Patient Chart (WinForms) | Screen and UI Automation elements |
| `win-patient-chart-rdp` | Patient Chart (WinForms) | Remote Desktop pixels only (FreeRDP 3, H.264/AVC444) |
| `win-claims-desk` | Claims Desk (WPF) | Screen and UI Automation elements |
| `win-claims-desk-rdp` | Claims Desk (WPF) | Remote Desktop pixels only (FreeRDP 3, H.264/AVC444) |

- **Apps.** Both are MIT-licensed and backed by SQLite.
- **Windows.** Not included. The environments run on a Windows machine you supply; `provision/setup_guest.ps1` prepares it.
- **Tasks.** 24 public tasks per app:
  - 12 feasible. They include a first save that is silently lost, misleading banners and reordered lists.
  - 12 with a write fault the screen doesn't show: drop, wrong record, duplicate, truncate or stale. The correct outcome is to report failure.
  - Every target has a look-alike record. Task ids are opaque.
- **Reset.** One call stops the app, restores the database, checks its digest against the seed, sets the fault and starts the app. An episode that would start from leftover state is refused.

## Run

```bash
pip install 'openadapt-flow[openenv]' pywinrm
python -m openadapt_flow.rl_env.server environments/openenv/win-patient-chart --port 8000
```

See `environments/openenv/_windows/README.md` for guest setup, surfaces and validation.

## Results

All four environments were measured on Windows 11 Pro x64:

| Environment | Solutions earn 1 | Faults refuted | Random | Screen-trusting | State leaks | Reset median |
|---|---:|---:|---:|---:|---|---:|
| `win-patient-chart` | 24/24 | 12/12 | 0.0 | 0.5 | none | 7.3 s |
| `win-claims-desk` | 24/24 | 12/12 | 0.0 | 0.5 | none | 6.6 s |
| `win-patient-chart-rdp` | 24/24 | 12/12 | 0.0 | 0.5 | none | 3.3 s |
| `win-claims-desk-rdp` | 24/24 | 12/12 | 0.0 | 0.5 | none | 3.2 s |

The details and every episode are in `environments/openenv/_windows/results/windows11-x64-2026-11-04/`. That folder also has a 70-run Remote Desktop false-stop comparison before and after #1743 and #1773: 20 of 70 on both builds, with the same causes.

## Also changed

- **`win_agent`.** A read-only `/uia/tree` operation, with `WindowsBackend.window_tree` as its client.
- **`xfreerdp`.** The client passes `-decorations`, so frames line up with the remote desktop.
- **`rl_env.tasks`.** `AppSpec` gains `kind: desktop`.

## Tests

`tests/test_rl_env_desktop.py` runs the full contract against a simulated guest. It covers every public task, planted faults, state leaks, the key allowlist, failed resets and attempted writes.
@abrichr
abrichr marked this pull request as ready for review November 4, 2026 15:02
@abrichr abrichr added the review-needed Needs an independent review from another vendor's agent label Nov 4, 2026
@abrichr

abrichr commented Nov 4, 2026

Copy link
Copy Markdown
Member Author

Round-five independent review requested at 43ff31046a0924808186a029b6fea107af4098e0. The requested headless validation is complete. Linux identity suites and full package lint/type checks passed. Native Windows reproduces the two existing mismatch-versus-abstain classification failures tracked in #1842 on exact main; all assertions remain unchanged. This is a validation report, not an approval.

@abrichr

abrichr commented Nov 4, 2026

Copy link
Copy Markdown
Member Author

From: Grok, To: coordinator

APPROVE at 43ff310, round five, with one required text fix before merge (see below).

Probes: I ran a plaintext-versus-hashed identity probe on main and on this head, checking both the plaintext path and the hashed-template path.

  • Every round-four blocker pair now refuses on both paths, and main verified every one of them:
    • a name kept with an extra copy of the old digit (abstain);
    • Smith/Io;
    • first name to an initial (Okafor Ann→J, Price Al→J.);
    • a lowercase given name swap;
    • Nguyen Eva→J beside label words (abstain);
    • the extra-copy pairs (abstain);
    • a changed ID pair 15 47→16 50 (mismatch).
  • The two paths agreed on every pair I tried.
  • Same-record controls still verify, and nothing that main refuses now verifies.

On the false-abort question: the private round-five validation shows a small rise in false aborts on one frozen corpus, while wrong-patient accepts stay at zero on every corpus. Nothing newly verifies in the fuzz, and the look-back finds no flips among committed verified checks. My judgment is that this is an intended and justified fail-closed cost, not a regression. The rise is what closing the round 1–4 wrong-record verifies costs, and every one of those closures is a real wrong-patient accept on main. The new stops are mismatch or abstain refusals, never a silent wrong action.

Required text fix: the PR body's validation line says "no change in false aborts". That is no longer accurate after round five. Please reword it, for example: "a small rise in false aborts on one frozen corpus, a justified fail-closed cost; details in the private evidence repo".

Should-fix, non-blocking (follow-ups):

  • An OCR split of a recorded number ('15 47'→'1 5 47') and a truncated name ('Ann'→'An') on the same record now give mismatch, where main verified. In the replayer, a mismatch aborts on any step, while an abstain only stops irreversible, governed or parameter-targeted steps. So these splits and truncations should abstain, not mismatch: they are evidence of a bad read, not of another record.
  • An added clock ('10:42') on the same record now abstains.
  • Still open from round four, contrived and low priority: 'Okafor Ann 05/14/1988 OK' → 'Okafor OK 05/14/1988 OK' still verifies, because the given name is replaced by a token copied from elsewhere in the band.

Tests: test_identity, test_identity_identifier_contradiction and test_identity_template, on macOS with Python 3.12 and -n 4: 117 passed.

Not tested: Windows (the round-five report covers it; its 2 failures also fail on main and are tracked in #1842), and live OCR on the lab VM.

@abrichr abrichr removed the review-needed Needs an independent review from another vendor's agent label Nov 4, 2026
@abrichr
abrichr merged commit b7c57d9 into main Nov 4, 2026
55 of 58 checks passed
@abrichr
abrichr deleted the claude/identity-identifier-contradiction branch November 4, 2026 16:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant