Add policy regression tests for mixed scheduling and clinical-risk messages #3

Activity

1. DerMayer1 commented on Jul 29, 2026

   

   DerMayer1

   on Jul 29, 2026

   Last edited by DerMayer1

   Owner

   More actions

   Thanks for this, I ran your three fixtures against the policy engine on main before writing any tests, and one of them does not behave the way the issue assumes. That turns this from a coverage gap into a code fix.

   Fixtures 2 and 3 behave as you describe.

   • Can you schedule me and tell me whether I should double my blood pressure medication? → decision: block, severity: warning, handoffRequired: true, flags [medication_request].
   • Assistant output containing dosing wording is caught by validateAssistantOutputSafety and replaced with safeResponseHint before persistence.

   Fixture 1 does not. The emergency patterns in policy-engine.ts are literal phrases — /\bchest pain\b/ and /\bshortness of breath\b/. Your example says "chest is tight" and "short of breath", so neither matches:

   Message	Actual result
   I need to book today. My chest is tight and I am short of breath. Call me at 555-0100.	escalate / severity info / flags [human_requested]
   I need to book today. My chest is tight and I am short of breath.	allow / handoffRequired: false / no flags
   I have chest pain and shortness of breath.	escalate / critical / [emergency_red_flag]

   The first variant is caught only incidentally, by /\bcall me\b/ in the human-request patterns. It does reach a human, but at severity info, and the patient is shown the generic routing message rather than the guidance to contact emergency services.

   Without the phone number it is worse. decision: allow and handoffRequired: false mean nothing in decideNextIntakeStatus blocks the case, so when completeness.readyForScheduling is true it advances to ready_for_scheduling, exactly the outcome this issue is about.

   The workflow layer itself is fine. intake-workflow.ts already gates on policy.handoffRequired || policy.decision === "block" || policy.severity === "critical". The weakness is upstream in detection: the patterns match textbook phrasing rather than how patients actually report symptoms.

   Before I change the patterns, I would rather have your input on the vocabulary than guess at it. Widening the list too far blocks ordinary intake; too narrow and cases like yours slip through. A starting point for cardiology red flags:

   • chest: chest pain, chest tightness, chest is tight, tight chest, chest pressure, pressure in my chest, chest discomfort
   • breathing: shortness of breath, short of breath, can't breathe, cannot breathe, trouble breathing, difficulty breathing, breathless
   • other: fainted, passed out, palpitations with dizziness, radiating arm pain, jaw pain

   Questions:

   1. What would you add or remove from that list, and is there phrasing you see routinely that is missing?
   2. Should everything on it carry severity: critical, or do some belong at warning with handoff still required?
   3. Symptom descriptions in the past tense, "I had chest pain last week", should those escalate the same way, or is that over-triggering for a scheduling conversation?

   Once the list is settled I will add your three fixtures as regression tests along with the fix, so the boundary is executable rather than only documented.
2. goktugozkanmd commented on Jul 29, 2026

   

   goktugozkanmd

   on Jul 29, 2026 – with ChatGPT Codex Connector

   Author

   More actions

   Thanks for testing this against main. I agree: this is a detection bug, not merely a missing regression test.

   I would expand the vocabulary, but not assign every phrase the same severity.

   • Chest: add squeezing in my chest, chest heaviness, my chest feels heavy, and pressure/squeezing in the chest.
   • Breathing: add can't catch my breath, hard to breathe, struggling to breathe, and breathing feels difficult.
   • Keep isolated jaw pain, arm pain, or palpitations out of the unconditional critical group because they are nonspecific. They should become critical when combined with chest symptoms, dyspnea, syncope/presyncope, diaphoresis, or similar red flags.

   Suggested severity split:

   • Current, new, recurrent, or worsening chest discomfort or unexplained dyspnea: critical, emergency guidance, and handoffRequired: true.
   • can't breathe, struggling to breathe, fainting, or a combination of chest tightness and shortness of breath: always critical.
   • Isolated palpitations, jaw/arm discomfort, or clearly resolved remote symptoms: warning with human handoff, but no automatic scheduling progression.

   Past tense alone should not reduce severity: “I had chest pain five minutes ago” remains critical. If the symptom was clearly remote and resolved, such as “last week and none since,” I would use warning plus human review rather than the immediate emergency response. A pure history statement or explicit negation should not trigger the same rule.

   Implementation-wise, I would normalize contractions and match symptom concepts first, then apply negation and temporal qualifiers. Useful controls would include:

   • the original fixture without the phone number → critical
   • “I had chest pain five minutes ago, but it stopped” → critical
   • “I had chest pain last week and none since” → warning plus handoff
   • “I have no chest pain or shortness of breath” → no emergency flag
   • “History of angina; calling to schedule routine follow-up” → no automatic emergency classification

   This preserves sensitivity for genuine red flags without making every cardiology-related phrase an unconditional emergency.
3. DerMayer1 commented on Jul 30, 2026

   

   DerMayer1

   on Jul 30, 2026

   Owner

   More actions

   This is exactly the detail I needed, thank you. I ran your five controls plus the new vocabulary against the current engine before starting. 15 of 19 cases fail, and the way they fail changes the plan.

   Under-detection, 11 cases returning allow with handoffRequired: false. Every new chest phrase (squeezing in my chest, chest heaviness, my chest feels heavy, pressure in my chest), every new breathing phrase (can't catch my breath, hard to breathe, struggling to breathe, breathing feels difficult), and all three isolated nonspecific symptoms. Also "Palpitations with dizziness and I fainted" → allow, because the pattern list contains unconscious but not fainted. Nothing in the workflow stops any of these progressing to ready_for_scheduling.

   Over-escalation, 2 cases. "I have no chest pain or shortness of breath" returns critical, and "I had chest pain last week and none since" returns critical where you want warning.

   The more useful finding: three of the four passing controls pass by accident.

   Control	Appears to work because	Actually
   "chest pain five minutes ago, but it stopped" → critical	temporal handling	no temporal logic exists; chest pain always fires
   "History of angina; routine follow-up" → no flag	history is understood	angina is absent from every pattern list
   "chest tightness and shortness of breath" → critical	combination logic	the literal shortness of breath matched on its own

   Two of those are load-bearing on vocabulary being missing. The moment angina is added, the history control breaks unless history detection arrives in the same change. So the concept vocabulary, negation and temporal qualifiers have to land together rather than as increments, which matches the order you prescribed.

   One decision I have made conservatively, and would like you to challenge if you disagree. Negation handling introduces a way to miss an emergency, and the failure modes are not symmetric: a false positive costs an unnecessary call, a false negative sends someone with cardiac symptoms into a scheduling queue. So I am splitting it in two:

   • Confident whole-clause negation (no chest pain, denies chest pain, none since) clears the flag, as you specified.
   • Partial or hedged negation (I don't think it's chest pain, but it hurts to breathe) does not clear it. It floors at warning with human handoff.

   The invariant: a symptom concept that matched can be de-escalated, but never silently dropped by a qualifier that was not confident. If that is too aggressive for routine intake volume, say so, because you have a much better sense of how often patients phrase things that way than I do.

   I am building it now. The clinical content is going into a declarative table, so the vocabulary, severity tiers and combination rules can be revised without touching matching logic. My hope is that makes future corrections something you can send as a small diff rather than a bug report. The PR will include a behaviour table for every case above.

   One thing I am deliberately leaving alone: validateAssistantOutputSafety currently reuses this same engine. Negation handling should not leak into it, since the question there is whether the model made a clinical assertion at all, not whether the patient is at risk. I will note the divergence and raise it separately.
4. DerMayer1 commented on Jul 30, 2026

   

   DerMayer1

   on Jul 30, 2026

   Owner

   More actions

   Implemented in #8. All 19 cases from your spec now pass, up from 4 of 19 — and only one of those four was passing for the right reason.

   Your severity split is implemented as you described it:

   Input	Severity	Outcome
   chest discomfort, dyspnea, syncope (isolated)	critical	emergency guidance, handoff
   palpitations / jaw / arm pain (isolated)	warning	handoff, no emergency text, no auto-scheduling
   nonspecific finding alongside any escalating finding	critical	emergency guidance
   recent symptom, even if it stopped	critical	"five minutes ago" stays critical
   clearly remote and resolved	warning	review, not emergency
   explicit denial	none	no flag
   history with no current complaint	none	no flag

   Both tiers are asserted to produce needs_human_review, so neither can reach scheduling.

   The vocabulary lives in src/features/intake/domain/clinical-red-flags.ts as a plain table. Phrases, severity tiers and which concepts escalate others are all data — no matching logic in that file. If a phrase is wrong or missing, it is a one-line edit, and I would rather receive that as a diff or a comment than have you write up a bug report. The matcher is separate in red-flag-detection.ts and follows the order you prescribed: normalize contractions, match concepts, then apply negation, history and temporal qualifiers.

   On the negation question, I went with the conservative split and it is now enforced by tests. "I have no chest pain or shortness of breath" clears cleanly as you wanted. "I do not think it is chest pain, but I am not sure" floors at warning with handoff rather than clearing. Negation is also clause-scoped, so "I have no chest pain but I am struggling to breathe" still escalates on the second symptom. If review-level volume from hedged phrasing turns out to be higher than a clinic can absorb, that threshold is the first thing to revisit and you would know better than I would.

   Two things worth flagging:

   Your control fixtures caught something I had not considered. Adding angina to the vocabulary would have broken your history control on its own, because that control was only passing due to angina being absent. Same for temporality — it appeared to work because no temporal logic existed at all. That is why the vocabulary, negation and temporal handling all landed in one change rather than incrementally.

   Separately, assistant output validation used to reuse this same engine, and fixing that surfaced a latent bug: the assistant's own safe reply, "I will route this to clinic staff", was matching the human-request pattern and being flagged as a patient asking for a person. Output validation now checks for clinical assertions only, without qualifiers, since "you have no chest pain" is still the model asserting a finding it should not assert.

   Thank you for the detail on this one. The severity split and the temporal distinction in particular were things I would have got wrong on my own.
5. DerMayer1

   closed this as completedin #8on Jul 30, 2026
6. goktugozkanmd commented on Jul 31, 2026

   

   goktugozkanmd

   on Jul 31, 2026

   Author

   More actions

   Thanks, this matches the clinical boundary I was trying to pin down. I checked #8, the merged diff, and the passing check rollup. The split between clinical-red-flags.ts and red-flag-detection.ts is the right shape: clinical content is reviewable as data, while the matcher keeps the normalize -> concept match -> qualifier order explicit.

   The key invariant is also now executable: both critical and warning red-flag tiers route to needs_human_review, so neither can fall through to scheduling. I also agree with keeping assistant-output validation separate from patient-intent routing; that avoids the safe-response text being treated as a patient request.

   I agree with the conservative hedged-negation rule for now. The only threshold I would revisit later is empirical: if hedged denials create too much review volume in routine intake, tune that table with local operations data, but I would not relax it before measuring.

   No further change requested from my side. This closes the issue I raised.