Skip to content

progress.md

Latest commit

 

History

History
714 lines (638 loc) · 161 KB

progress.md

File metadata and controls

714 lines (638 loc) · 161 KB
Raw

progress.md — Parallax live state

Single source of truth for what is happening right now. Every session reads this after CONTRIBUTING.md and plan.md. Rules: edit only your own status rows and your own role section. The shared log at the bottom is append-only: add new entries at the very end, never edit old ones. Format is in plan.md §12.3. .gitattributes sets merge=union for this file.

Team

Role Person Machine / GPU Groq key variable
P1 Imaging Core GROQ_KEY_AUDIO
P2 Data, Training & Validation laptop, RTX 4050 6 GB GROQ_KEY_JUDGE
P3 Clinical Reasoning & Trust GROQ_KEY_REPORT
P4 Experience & Platform GROQ_KEY_EXTRACT

Setup (plan.md §6)

  • Everyone: repo access, git identity set
  • Everyone: own Groq key in local .env
  • Everyone: git config core.hooksPath .githooks (hook verified to reject assistant mentions)
  • P2: Kaggle token; RSNA rules accepted; BDNeuro-MRI downloaded
  • P3: HF token; MedGemma 1.5 + MedSigLIP terms accepted; Kaggle phone-verified
  • P4: skeleton committed; make setup green on a fresh clone
  • Groq /models checked at T+0; IDs in config match (record below)
  • Local NVIDIA GPU available? (who, VRAM):

Gates

Gate Target time Criterion Status Notes
G0 T+1 Contract frozen, fixtures committed todo
G1 T+4 CXR end to end with real output todo
G2 T+7 All 4 modalities end to end; status rule + firewall live todo
G3 T+9 Metrics frozen in reports/ todo
G4 T+11 Feature freeze todo
Submit T+13 Repo link submitted via form; backup video recorded todo

Status board

States: todo · doing · blocked · done · cut

P1 — Imaging Core

WP Task State Branch Updated Note
P1.1 Decode PNG/JPG/DICOM + hashing done p1/intake 2026-10-06 intake/decode.py, intake/preprocess.py
P1.2 DICOM PHI scrub done p1/intake 2026-10-06 intake/phi.py
P1.3 Quality gate done p1/intake 2026-10-06 thresholds provisional, refit on real data
P1.4 Router (MedSigLIP probe) done p1/intake 2026-10-07 real MedSigLIP; held-out n=50 acc 1.00 (CI 0.93 to 1.00); zero-shot 95.6% on 320; classes come from separate datasets, so the 97% mixed-set number is P2's; body-part head unvalidated
P1.5 OOD score done p1/intake 2026-10-07 Mahalanobis per modality, AUROC 1.0 on held-out (small n); energy score tested and rejected (AUROC 0.05 on chest); thresholds provisional
P1.6 CXR reader + anatomy zones done p1/intake 2026-10-07 67 reader tests incl. 5 on real weights; localization quality not yet benchmarked (P2.7)
P1.7 Bone reader wrapper done p1/intake 2026-10-07 trained bone_det weights (hash matches model_meta.json) load through BoneReader.from_dir; real boxes on staged FracAtlas films (scores 0.58 and 0.74 on two fractures, near 0 on non-fractures); check_trained_models.py ok
P1.8 Brain + skin inference wrappers done p1/intake 2026-10-07 trained brain_cls, brain_seg and skin_cls load and run; Grad-CAM++ boxes, U-Net tumour masks (0.6-4.8% of slice), faithfulness and stability measured per model (results/faithfulness_trained.json); MedSAM masks on the trained skin classifier's CAM boxes checked on a staged ISIC image (real MedSAM weights, masks of about 47,000 px, no flags)
P1.9 Faithfulness test (D1) done p1/intake 2026-10-07 chest: 0/18 findings pass on 12 films, random regions 0/18. Trained models on staged images: brain 42%, skin 44% pass (heatmap test), bone 6 of 7 fracture findings pass (new box-region test, equal-area random boxes)
P1.10 Saliency sanity check (D2) done p1/intake 2026-10-07 chest table committed; gradcam++ chosen, gradcam and hirescam collapse to empty maps; single film, repeat on more; other models wait for weights
P1.11 Stability score (D5) done p1/intake 2026-10-07 validated on real chest model; flip rate 0 at severity 2 on the sample film, severity 5 flips contrast
P1.12 MedSAM service done p1/intake 2026-10-07 real MedSAM on 30 ISIC 2018 validation lesions: mean Jaccard 0.878, Dice 0.934, 0 failures below 0.65; CPU about 30 s per image

P2 — Data, Training & Validation

WP Task State Branch Updated Note
P2.1 Dataset downloads + manifests done main 2026-10-07 all script-fetchable sets present with manifests and a pinned lock file; BDNeuro registered from the manual download
P2.2 Leakage audit + group splits done main 2026-10-07 all six datasets audited; reports/leakage.json
P2.3 Brain classifier (Kaggle) done main 2026-10-07 done on Kaggle (second account); registered
P2.4 Brain segmenter (Kaggle) done main 2026-10-07 done on Kaggle (second account); registered
P2.5 Skin classifier (Kaggle) done main 2026-10-07 official-test BMA 0.680 [0.632, 0.728]; registered
P2.6 Bone detector (Kaggle) done main 2026-10-07 mAP50 0.453, image AUROC 0.923; registered
P2.7 CXR calibration + external validation done main 2026-10-07 RSNA: chex AUROC 0.785, mimic_ch 0.749, contaminated all 0.875; pointing game and IoU in reports/cxr.json
P2.8 Temperature scaling + conformal done main 2026-10-07 backend/medproof/calibrate; skin official-test coverage 0.898 at the 0.90 target; chest and bone FNR within 3 points; brain over-covers (0.94)
P2.9 Selective prediction done main 2026-10-07 AURC and risk-coverage for skin, brain, chest, bone; reports/figures/risk_coverage.png
P2.10 Trust-signal validation (D13) done main 2026-10-07 unstable, abstain, OOD-energy predict errors; quality gate does not; discordant and unfaithful pending P3 and P1.9 (harness ready)
P2.11 Subgroup audit done main 2026-10-07 age, sex, site, body part, view; worst subgroups named in reports/subgroups.json
P2.12 Corruption benchmark done main 2026-10-07 8 perturbations x 5 severities on 5 models; reports/corruption.json and figure
P2.13 Model cards + datasheets done main 2026-10-07 docs/model_cards (8) and docs/datasheets (7), generated from metrics.json
P2.14 make eval done main 2026-10-07 python ml/eval/run_all.py: about 100 s, byte-reproducible; reports/metrics.json is the single source

P3 — Clinical Reasoning & Trust

WP Task State Branch Updated Note
P3.1 MedGemma service + batch reads doing p3/medgemma-groq 2026-10-07 real read verified on the local RTX 4060 (bf16 4-bit, 3.2 GiB, 5.5 s/read after a 23 s load); FracAtlas batch complete (569 reads); HAM10000 batch complete (1,000 reads, run on a new CUDA venv .venv-medgemma); Kaggle/Colab notebooks untested; CXR box format still not verified
P3.2 Generalist reader + concordance done p3/medgemma-groq 2026-10-07 reports/concordance.json final for both batches; concordance now matches vocabulary names and dataset codes, and a disagreement only demotes a finding in modalities listed in DOWNGRADE_VALIDATED (none yet: bone flags 83% of correct reports, skin effect not significant)
P3.3 Groq pool done p3/medgemma-groq 2026-10-07 27 tests with a mocked Groq; live call confirmed on gpt-oss-20b and 120b; Prompt Guard score() added
P3.4 Synthetic note generator done p3/medgemma-groq 2026-10-07 ml/data/gen_notes.py, 200 notes: 60 contradictions, 48 attacks in 8 families, 12 benign look-alikes, clean twins; notes_v1.jsonl committed
P3.5 Fact extraction with spans done p3/medgemma-groq 2026-10-07 exact-span F1 0.919 on 62 held-out notes (target 0.85); cue scan added for statements the model skips; end-to-end numbers in reports/p3_end_to_end_test.json
P3.6 Injection guard done p3/medgemma-groq 2026-10-07 regex + Prompt Guard + LLM layers; 27 of 28 fresh attacks with all three layers (1 false positive in 15 benign), so the 100% target is not met on fresh attacks; two regex phrasings found by red-teaming are now covered
P3.7 Contradiction rules done p3/medgemma-groq 2026-10-07 P/R 1.00/1.00 on gold facts (templated notes, so it shows the rules fire, not that they generalise); end-to-end with extracted facts not yet measured
P3.8 Retrieval / precedents done p3/medgemma-groq 2026-10-07 MedSigLIP indices for all four modalities (train-only, or the calibration split for chest; own id and group excluded). Precision@5 against chance: FracAtlas 0.84 (0.70), HAM10000 0.67 (0.47), brain 0.92 (0.27), chest 0.67 (0.36); majority-vote balanced accuracy brain 0.95, chest 0.72, HAM10000 0.49. Precedent stage covers all four; the weakest class everywhere is the rare one (fracture 0.48, melanoma 0.43, chest opacity 0.57)
P3.9 Slot-filled report done p3/medgemma-groq 2026-10-07 slot renderer, frames, model-chooses-frame drafts with offline template fallback; live gpt-oss-120b call verified
P3.10 Hallucination firewall done p3/medgemma-groq 2026-10-07 R1-R12; 100% of 100+ planted bad claims blocked, 0 of 10 good claims blocked; R5 treats faithful=None as unverified until P1.9
P3.11 Entailment judge done main 2026-10-07 60 labelled sentences: first pass 0.917; after fixing what the judge is shown, 59/60 on a fresh variant (kept 29/30 true, caught 30/30 defective) and 60/60 on the tuned one; judge outage leaves claims explicitly unchecked
P3.12 Voice (Whisper) done main 2026-10-07 real round trip: synthesized dictation -> Whisper on Groq -> guard -> facts; spoken injection flagged and kept out of the facts
P3.13 FHIR export done main 2026-10-07 collection Bundle (DiagnosticReport + Observations) validated with fhir.resources R4B; quotes redacted unless requested

P4 — Experience & Platform

WP Task State Branch Updated Note
P4.1 Repo skeleton + Makefile + tooling done main 2026-10-07 Makefile, root + backend pyproject tooling, pre-commit, minimal web/ (Vite+React+TS)
P4.2 Contract + fixtures (G0) done main 2026-10-07 schemas.py unchanged (additive only); core/vocab.py, core/status_rule.py; JSON Schema + TS export; 4 fixtures
P4.3 Pipeline orchestrator done main 2026-10-07 pipeline.py + core/{context,config,cache}.py; only intake + CXR reader stages exist today, new ones append to PIPELINE
P4.4 API + SSE done main 2026-10-07 api/{app,studies,system}.py; FastAPI + sse-starlette; in-memory core/store.py (not persisted)
P4.5 Evidence ledger done main 2026-10-07 core/ledger.py; single global hash-chained JSONL; GET /ledger/verify; doctor feedback logged
P4.6 Web app pages done main 2026-10-07 Scrollytelling landing (prerendered, two Vite entries) + /read workstation + validation, models, audit, report, 404; Playwright + axe e2e green
P4.7 WebGL viewer done main 2026-10-07 Own WebGL2 engine web/src/gl/; R16F full-depth path, W/L, overlays, keyboard map. 60 fps on 2048² not measured on a real GPU yet (see role section)
P4.8 Docker + CI + e2e + a11y todo
P4.9 Offline demo pack todo
P4.10 README done main 2026-10-07 Submission README (what, pipeline, models, evidence, sample I/O, results with CIs, scope, install, configure, run, reproduce, licences); real chest run in demo/sample_output/; screenshots in docs/screenshots/

Role sections

P1 — Imaging Core

Now: every P1 package is implemented and validated on real models or data, including P1.7 and P1.8 on P2's trained weights (placed locally at ml/artifacts/<model>/weights.pt, git-ignored, hashes match model_meta.json). Next: refit quality, faithfulness and OOD thresholds on real validation sets; P4 to wire the router and the faithfulness and stability stages for brain, skin and bone. Blockers: none for P1.7 and P1.8. Still open for others: P2 must revoke the Kaggle token that was committed in .env (still in history); P2's test_eval_reproducible fails on this machine. Replies to P4's assigned work (P1-A to P1-E), 2026-10-07:

  • P1-A masks, done. readers.classifier.run(ctx, reader, unet=None, medsam=None) and readers.bone.run(ctx, reader, medsam=None) call segmenter.attach_masks before to_findings; the mask PNG goes under ctx.artifact_dir. Loader calls: unet = UNetSegmenter.from_dir("ml/artifacts/brain_seg") (from medproof.readers.segmenter), medsam = MedSAM.load() (from medproof.segment.medsam; it reads MEDPROOF_MEDSAM_MODEL, a local folder or the hub id). Brain: classifier.run(ctx, reader=brain_reader, unet=unet); skin: classifier.run(ctx, reader=skin_reader, medsam=medsam); bone: bone.run(ctx, reader=bone_reader, medsam=medsam). Load once at start-up, not per study. A failing mask model keeps the finding and adds a generic warning. Checked end to end on the real trained models: brain finding gets heatmap + U-Net mask, skin gets heatmap + MedSAM mask, bone gets bbox + MedSAM mask. Needs segmentation-models-pytorch (in the ml extra) on the demo laptop.
  • P1-B bone faithfulness, done. verify.faithfulness.assess_output now runs assess_box for findings with a box and no heatmap: the strongest box is blurred and the probability drop is compared with equal-area boxes elsewhere (same permutation rule: p <= 0.10 and drop >= 0.05). Boxes above 35% of the image are "not assessable". Real bone detector on the 7 staged fracture films: 6 faithful, mean drop 0.43. No change needed in P4's stage chain: the existing faithfulness stage fills faithful for bone findings.
  • P1-C brain heatmap, investigated and acted on. On the demo brain image the heatmap peaks at the left image border (x=0) while the U-Net puts the tumour at x 327-437; deleting the tumour box does not beat random boxes (p=0.6). So the classifier is not using the tumour there. Measured on 16 staged brain slices with a U-Net mask: Grad-CAM++, Grad-CAM and HiResCAM, final or earlier layers, all put only 6-14% of their salient pixels inside the tumour mask (14 of 16 below 0.2 for the default). No CAM variant fixes it, so it is the model, not the explanation method. New rule: when the U-Net mask is attached and less than 20% of the heatmap's salient pixels fall inside it, the finding gets the flag heatmap_off_target and its heatmap evidence is marked faithful=false (flag unfaithful) even if the deletion test passes, because a region the model uses but that is not the lesion cannot prove the finding. Provisional: the U-Net was trained on LGG FLAIR and is applied to a different dataset, so its masks are not validated on these slices.
  • P1-D router and OOD, ready to hand over. Verified from the exported folder: with MEDPROOF_ROUTER=1 and MEDPROOF_ROUTER_MODEL set, uploads without a hint are routed and ood appears (distance, threshold, score, is_ood). Files (git-ignored, copied by hand): backend/artifacts/router_probe.npz sha256 0f3042717e9230dd1be9c9e444a7b7c954115a48ef3f8f575099f07f44c97aba, backend/artifacts/ood.npz sha256 38d268cc045bfb56bd713abe751567283ebbbb2a1216dc2f0b5bf4ea0e07d8be. python scripts/fetch_models.py downloads MedSigLIP and MedSAM into backend/models/ with the machine's own HF token (the account must have accepted the MedSigLIP terms) and prints the two env lines; --from-cache copies from a machine that already holds them. A ready folder set is at medproof-kit/handover/ on P1's machine (medsiglip/, medsam/, both .npz).
  • P1-E MedSigLIP for precedents, same folder. MEDPROOF_ROUTER_MODEL points at it; P3's embedder can use the same folder. For P2 (available now on branch p1/intake):
  • Decode any upload: from medproof.intake.decode import load_image returns DecodedImage (.display uint8, .analysis float32 in [0,1], .sha256). DICOM handles rescale slope/intercept, window tags and MONOCHROME1 (RSNA). Raises DecodeError on bad input. Use this for pHash and for every eval loader so you see the pixels the product sees.
  • Training and serving preprocessing: from medproof.intake.preprocess import prepare, get_spec; specs cxr_xrv, brain_effnet, skin_cls, bone_yolo. Call prepare(decoded_or_float_array, "brain_effnet") to get a float32 CHW tensor. Change a spec only via a Decisions entry so train and serve stay equal.
  • Corruption benchmark: from medproof.verify.perturbations import apply, NAMES, SEVERITIES; apply(name, img_float01, severity 1..5, seed). Eight perturbations, deterministic for a given seed.
  • Quality gate: from medproof.intake.quality import assess, QualityConfig; assess(decoded, "cxr") gives .passed, .reasons (code, level, fix), .metrics. Thresholds are in QualityConfig and were set on a synthetic phantom; refit them on real validation images (target: at most 5% of clean images flagged).
  • Reader output shape: medproof.readers.base.ReaderOutput (final with P1.6).
  • P2.7 handoff (CXR reader, available on p1/intake and main):
    • Needs: pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu and pip install torchxrayvision. Weights download on first use to ~/.torchxrayvision/ (DenseNet 28 MB, anatomy PSPNet 273 MB).
    • Probabilities only: CxrReader.load(CxrConfig(weights="densenet121-res224-chex")).score(img) returns an (18,) float32 array aligned to reader.labels (empty-string labels are unsupported by that weight set, ignore them). For mimic_ch or chex external validation on RSNA use those weights; -all saw RSNA (contaminated). MEDPROOF_CXR_WEIGHTS overrides the default.
    • These outputs are NOT raw logits: torchxrayvision applies sigmoid then per-label operating-point scaling, so 0.5 is each label's operating threshold. Calibrate on these values as given and say so in the model card. ReaderOutput.output_kind says "probability".
    • Full read with localization: CxrReader.load(anatomy_fn=AnatomySegmenter.try_load()).predict(img) then evidence.to_findings(out, cfg, artifact_dir=...) gives contract Finding and ImageEvidence (prob_calibrated equals prob_raw, flag uncalibrated, conformal_set=[]: calibration fills these). Stage entry point: readers.cxr.run(ctx).
    • Localization eval: boxes are bbox_xyxy in original-image pixels (max edge exclusive); heatmaps are float32 [0,1] on the original grid. The model reads only the central square of non-square images; heatmap outside it is exactly 0 and the finding carries flag partial_view. RSNA images are square, so no effect there.
    • Preprocessing: intake.preprocess.prepare(img, "cxr_xrv") now centre-crops to a square then resizes to 224 (TorchXRayVision's own convention). The spec changed from stretch to centre-crop, so any notebook using it for CXR should re-run; brain, skin and bone specs are unchanged.
  • P1.4 to P1.12 handoff (all on p1/intake and main):
    • Router: intake.router.Router(embedder, config); run(ctx) stage returns payload = {modality, body_part, confidence, probs, method, flags, ood}. Needs pip install transformers scikit-learn and the gated google/medsiglip-448 (HF token in .env; or MEDPROOF_ROUTER_MODEL=<local folder>). intake.run_bytes(raw, router=...) routes before the quality checks. Artifacts are git-ignored; register these hashes: probe router_probe.npz sha256 0f3042717e9230dd1be9c9e444a7b7c954115a48ef3f8f575099f07f44c97aba (45 KB), OOD ood.npz sha256 38d268cc045bfb56bd713abe751567283ebbbb2a1216dc2f0b5bf4ea0e07d8be (21 MB). Both were trained on 320 public images; retrain with python -m medproof.intake.router_train --data DIR and python -m medproof.intake.ood --data DIR (DIR//... layout; scripts/stage_router_data.py builds a small one) on the full datasets for the final numbers.
    • P2, router and OOD evaluation: intake.router_train.evaluate_folder(embedder, DIR) scores only the held-out groups; intake.ood.score_folder(embedder, DIR, OODModel.load(path)) returns one row per image (distance, threshold, score, is_ood) for AUROC against other modalities and natural images. The 97% target is yours to report on the mixed held-out set; my staged set cannot show it (see Decisions).
    • Faithfulness: verify.faithfulness.assess_output(reader, decoded, reader_output) then apply_to_findings(findings, results) fills ImageEvidence.faithful and faithfulness_drop and adds flag unfaithful. Stability: verify.stability.assess_output then apply_to_findings fills the contract Stability and flag unstable (flip rate above 0.25). Both are model-agnostic: any reader with .score(img) works (CXR, classifier, bone).
    • P3: readers expose boxes as bbox_xyxy in original-image pixels (max edge exclusive), one ImageEvidence per finding plus a second mask-kind evidence when a U-Net or MedSAM mask exists; bone findings have no heatmap and use kind bbox.
    • P4 wiring recipe (the pipeline's reader step still says "no router available yet (P1.4)"): (1) when ctx.modality_hint is None, call intake.router.run(ctx) (it uses get_default_router(), which is off unless artifacts/router_probe.npz exists or MEDPROOF_ROUTER=1) and set the hint only when payload["confidence"] is at least RouterConfig().trust_confidence (0.60); copy the router flags (ood, router_uncertain) onto every finding's flags, since core/status_rule.py downgrades ood. (2) Reader: readers.cxr.run(ctx), readers.classifier.run(ctx, reader=ImageClassifierReader.from_dir(...)) or readers.bone.run(ctx, reader=BoneReader.from_dir(...)); each now leaves ctx.reader, ctx.reader_output and ctx.findings on the context. (3) Then verify.faithfulness.run(ctx, reader=ctx.reader) and verify.stability.run(ctx, reader=ctx.reader) return updated findings in payload["findings"] (faithful, faithfulness_drop, stability, flags). Budget: stability costs 8 forward passes (about 3 s on CPU for the chest model); faithfulness costs about 14 s per finding with the default 16 random placements (9 is the minimum), so run it on the top two or three findings or as a later refinement streamed over SSE.
    • P4: stage entry points are intake.run.run, intake.router.run, readers.cxr.run, readers.classifier.run, readers.bone.run, verify.faithfulness.run, verify.stability.run, segment.medsam.run_box; each returns a StageResult and never raises. backend/pyproject.toml gained ml and services extras; please fold them into your uv setup. Real-model tests skip with a reason unless the weights are on disk.
  • Setup: cd backend, Python 3.12 venv, pip install -e .[dev,ml,services] (CPU torch from https://download.pytorch.org/whl/cpu), then python -m pytest -q. Real-model tests need MEDPROOF_ROUTER_MODEL (local MedSigLIP folder), MEDPROOF_ROUTER_DATA (staged images) and the cached chest and anatomy weights.

P2 — Data, Training & Validation

Now: every P2 work package is done and pushed to main (P2.1 to P2.14). python ml/eval/run_all.py --check reproduces reports/metrics.json exactly in about 100 s from committed files only; docs/validation_report.md, 8 model cards, 7 datasheets and 6 figures are generated from it. Tests: python -m pytest ml/tests -q and cd backend && python -m pytest -q (555 passed). Next: nothing blocking. Open items that need other roles are listed below. Optional P2 latitude if time remains: MILK10k external skin test (M4), a higher-resolution skin model or ensemble (official-test balanced accuracy is 0.68), the discordant and unfaithful signals once their inputs exist. Blockers: none. BDNeuro-MRI licence is unconfirmed (README placeholder); the datasheet says so. Machine: laptop, NVIDIA RTX 4050 6 GB. For P1 (everything below is on main):

  • Weights are not in git. They are in ml/artifacts/<model>/weights.pt on the P2 laptop and in the private Kaggle outputs (rishijayanath/parallax-p2-6-bone-fracture-detector; johannshonigeorge/parallax-p2-3-brain-mri-classifier, parallax-p2-4-brain-tumour-segmenter, parallax-p2-5-skin-lesion-classifier). Ask P2 to send them, or pull with the Kaggle CLI. ml/artifacts/registry.json has the class order, preprocessing spec name, sha256 and licence for each.
  • Rebuild a classifier with timm.create_model(arch, pretrained=False, num_classes=len(classes)) and load_state_dict(torch.load(weights.pt)); the segmenter is smp.Unet("resnet34", encoder_weights=None, in_channels=3, classes=1); the bone model loads with ultralytics.YOLO(weights.pt) and predict(imgsz=640, conf=0.001) (the library letterboxes itself). The segmenter uses a new spec brain_seg (256 px, 3 channels, ImageNet mean and std, stretch): please add it to intake/preprocess.py (its definition is SEG_SPEC in ml/train/seg.py).
  • Calibration to apply at serving time, no torch needed: from medproof.calibrate.calibrator import Calibrator; Calibrator.load("ml/artifacts/skin_cls/calibration.json").calibrate(logits) returns calibrated probabilities, a conformal set of class names (target coverage 90%) and a tier (high/moderate/low/abstain) per input; same for brain_cls. Binary readers: BinaryCalibrator.load("ml/artifacts/bone_det/calibration.json") and ml/artifacts/cxr_chex/calibration_lung.json (.prob(score), .decide(score) with the FNR-controlled report flag and abstention band). The product demo reads the chest model all, which saw RSNA, so the calibrator fitted for it on RSNA is optimistic: either switch the product to the chex weights, or accept uncalibrated-looking confidence and say so.
  • Quality gate needs refitting. On real clean images it flags 43% (skin), 88% (brain), 45% (chest) and 91% (bone) of images, and flagged images are not wrong more often (reports/signals.json). Dominant reasons: clipped (black borders in MRI and chest films count as clipped pixels), low_contrast and overexposed for dermoscopy, low_resolution for small FracAtlas images. Percentile tables per metric are in the same file.
  • Truncated JPEGs. 59 FracAtlas negatives lose their last few bytes; load_image raises DecodeError on them although OpenCV and tolerant PIL decode them fine. A tolerant fallback in intake/decode.py would keep such uploads working.
  • fp16 inference on the brain classifier produced non-finite logits for one unusual image; serving on CPU float32 is unaffected, but guard against non-finite outputs.
  • Skin ensemble (optional, recommended). Weights for the two extra seeds are ml/artifacts/skin_cls_s2/weights.pt and skin_cls_s3/weights.pt (P2 will send D:\\parallax_weights_ensemble.zip). Serving: average the three models' logits (optionally over four flips of the preprocessed tensor), then Calibrator.load("ml/artifacts/skin_ens/calibration.json"). It costs three times the weights and inference time for about +0.023 balanced accuracy; keep the single skin_cls if latency matters.
  • Faithfulness (P1.9) and stability (P1.11): ml/eval/signals.py::compare(df, flag) validates any per-image boolean column against errors. The corruption benchmark already shows flip rate at severity 2 above 0.25 predicts errors strongly (skin 0.61 vs 0.13 error; brain 0.87 vs 0.03). For P3: specialist predictions on the same images as your MedGemma batches are cached: ml/artifacts/skin_cls/predictions/official_test.npz (ids, logits, y, group) and ml/artifacts/bone_det/predictions/test.npz (ids, packed boxes). Once the reads are complete, build a per-image discordant column and call signals.compare; a bone discordant flag must not downgrade findings until it passes. The chest reader is validated in reports/cxr.json. For P4: reports/metrics.json is the only file the validation page needs (headline rows with CIs and contamination status, calibration, selective prediction, trust signals, subgroups, corruption, OOD, leakage, registry); docs/model_cards/index.json and docs/datasheets/index.json list the cards and sheets for the /models page; reports/figures/*.png are static fallbacks. ml/Makefile.inc defines data, eval, eval-check, figures, ml-test: please include it and ack the .gitignore carve-out for ml/data/ (Decisions, 2026-10-06). Kaggle jobs: (notebook slug, started, status, artifact hash)
  • rishijayanath/parallax-p2-5-skin-lesion-classifier: 2026-10-07 01:00 to 01:17, COMPLETE. Official ISIC test balanced accuracy 0.689 [0.640, 0.738]; output not pulled (Kaggle rate limit), superseded by the rerun below
  • rishijayanath/parallax-p2-6-bone-fracture-detector: v1 trained 80 epochs then failed at prediction export on truncated JPEGs (fixed in 8539baf); v2 COMPLETE 2026-10-07 02:56, pulled, registered as bone_det@8539baf9. Test (569 images, 103 fractured): mAP50 0.453 [0.367, 0.547], image AUROC 0.923 [0.886, 0.957], sensitivity 0.835 [0.757, 0.903] at the validation-fixed 90%-specificity threshold (test specificity 0.880)
  • johannshonigeorge/parallax-p2-5-skin-lesion-classifier: rerun COMPLETE 2026-10-07 02:26, pulled, registered as skin_cls@a217517b. Official ISIC 2018 test balanced accuracy 0.680 [0.632, 0.728], melanoma recall 0.637, melanoma AUROC 0.924; the two runs differ by about 0.01 (GPU nondeterminism)
  • johannshonigeorge/parallax-p2-3-brain-mri-classifier: COMPLETE 2026-10-07, pulled, registered as brain_cls@d351f062. Leakage-free test accuracy 0.940 [0.921, 0.958]
  • johannshonigeorge/parallax-p2-4-brain-tumour-segmenter: COMPLETE, pulled, registered as brain_seg@d351f062. Mean per-patient Dice 0.831 [0.705, 0.891] over 9 test groups. A duplicate of this job also ran on rishijayanath (the launcher fired before I stopped it); its output is unused

P3 — Clinical Reasoning & Trust

Now: Everything that needs only P3 is done and committed: pipeline stage adapters (medproof/reasoning_stages.py: second_read, context, report, precedents), the discordance policy, label canonicalisation, lexicon derived from core/vocab.py, a third injection set, demo second reads in demo/medgemma_reads, and a 96-case red-team suite. The adapters wait on a small orchestrator change (Contract change request P3-1). Next: once P4 acks P3-1, register stage_specs() and check a live study end to end; brain and CXR retrieval, MedGemma box quality on RSNA and any CXR/brain second-read batch when P2's images arrive; Kaggle/Colab notebooks need a manual GPU session; M1/M2 only if time remains. Blockers: (1) Real faithful flags for brain, skin and bone: P4 and P1 still need to register those readers in pipeline._reader_step and wire the faithfulness and stability stages. (2) Nothing else on the P3 side is waiting on another role; the RSNA work was finished by P3 (see the log) and the earlier handoff to P2 is withdrawn. MedGemma endpoint: local only. uvicorn services.medgemma.server:app --port 8001 on the RTX 4060 (nf4 weights, bf16 compute, 3.2 GiB VRAM, about 5.5 s per read after a 23 s load). Kaggle/Colab notebooks written but not run. For P1: report/firewall.py::is_supported treats faithful=None as unverified; set faithful (P1.9) and R5 tightens automatically. readers/generalist.py reads DecodedImage.display and .sha256; verify/concordance.apply(findings, read) fills Finding.second_read and adds flags only, never status. For P2 / P4 (important): the second reader is weak on bone: on a random 403-image prefix of the FracAtlas batch MedGemma 4B (4-bit, zero-shot) found about 19% of fractures (sensitivity 0.19, 95% CI 0.11-0.29) at 99.4% specificity (kappa 0.27, 0.15-0.38); the missed reads literally say "No obvious fractures are visible". A discordant status driven by second_reader_disagrees would downgrade most true bone positives. Per plan 9.4, validate the flag (D13) per modality before letting it downgrade anything; until then treat it as an audit flag for bone. Numbers will be in reports/concordance.json. For P2 (D13 input, ready): reports/concordance.json has datasets.<name>.specialist_vs_medgemma.discordant_signal (computed with ml.eval.signals.compare). Bone: the specialist is wrong far more often when MedGemma disagrees (0.63 vs 0.02), but 83% of the detector's correct fracture reports are also flagged, because MedGemma names a fracture on only about 4% of images. So the flag predicts errors and still must not downgrade bone findings. Skin (1,000 reads, 420 where MedGemma named a lesion type): error rate 0.21 flagged vs 0.15 unflagged, difference 0.057, CI -0.017 to 0.125, not significant, so do not downgrade on it. Side note: signals.compare raises IndexError when no unflagged case is wrong (the risk-ratio bootstrap is all-NaN); my wrapper catches it, but you may want to guard it too. RSNA work (done by P3, no action needed from P2): the images were downloaded with python ml/data/download.py rsna_pneumonia (needs pip install kaggle, KAGGLE_API_TOKEN, the competition rules accepted on Kaggle, and PYTHONUTF8=1 on Windows). The chest precedent index (ml/artifacts/retrieval/rsna_medsiglip-448.{json,npy}, git-ignored) and the box test were built from them; results are in reports/retrieval_rsna_medsiglip-448.json and reports/medgemma_boxes_rsna_{optout,forced}.json. P4: the precedents stage needs the index file on the server machine or MEDPROOF_RETRIEVAL_DIR; ask P3 for the 35 MB .npy. Answers to P4's four requests (P3-A, P3-C, P3-D, P3-E), 2026-10-07:

  • P3-A, precedent indices and thumbnails. All four indices exist (skin, bone, chest and brain) and are packaged as retrieval_pack.zip (127 MB, kept off git; the folder layout, hashes of the index files, licences and how to install are in docs/retrieval_indices.md). Unzip into ml/artifacts/retrieval/ or set MEDPROOF_RETRIEVAL_DIR. Thumbnails: FracAtlas yes (CC BY 4.0, credit it), brain yes too (CC0 as stated; it merges sources with their own terms, so credit it), HAM10000/ISIC no (CC BY-NC), RSNA no. That is enforced in code, not by convention: a precedent from skin or chest always has thumb_ref == "" and shows label and similarity only. For bone and brain the stage copies the thumbnail into the study's artifact folder and sets thumb_ref to /studies/{id}/artifacts/precedent_<dataset>_<case>.png, which your artifact route already serves (PNG, which is why they are PNG). MedSigLIP comes from google/medsiglip-448 with an accepted-terms HF_TOKEN; the precedents stage needs only its image tower, so it does not hit the tokenizer problem the router has. Warm it at start-up (first call 10 to 45 s on a CPU).
  • P3-C, voice dictation. transcribe_dictation and process_dictation are final, and the audio key works: a synthesized 9 s dictation went through Whisper on Groq with GROQ_KEY_AUDIO today ("Sixty two year old woman ..." came back as "62-year-old woman ..."). The transcript appends to the typed note, it does not replace it. Use from medproof.context.voice import dictation_to_note; dictation_to_note(audio_bytes, filename, typed_text, pool) returns .ok, .note (typed text, a newline, then the transcript; just the typed text if transcription failed), .transcript, .language, .duration_s and .warnings, and never raises. Suggested flow: a small POST /transcribe that takes the audio and the current notes text and returns .note, which you put back into the notes box so the doctor can read and correct it before pressing Analyse; the audio itself never enters the study. It is one note (n1), so the guard, the fact spans and the highlighting cover the dictated part: a spoken "ignore previous instructions" is quarantined like a typed one (tested). Allowed types: wav, mp3, m4a, mp4, mpeg, mpga, ogg, flac, webm (a browser MediaRecorder makes webm); limit 25 MB.
  • P3-D, should a failed or untestable region cap a finding at uncertain? Decision: a failed region is withheld; an untestable region caps the finding at uncertain; only a region that passed the test can make a finding verified. (1) Failed (faithful is False): already handled on my side: the context stage no longer lets a fitting symptom count as support for it, so the finding is rejected with no claim (this is what withholds most brain findings). (2) Untestable (faithful is None, for example a chest finding outside the top three that get tested): notes may raise it to uncertain but never to verified. That is your one-line change, in core/status_rule.py, inside _is_uncertain (see Contract change request P3-2). (3) Bone is no longer untestable: with P1's box-region test the demo fracture passes (faithful True) and is verified on the image, so the bone case you saw (verified on note support alone) is out of date.
  • P3-E, report wording. Fixed. The sentence that cites the note now keeps the region ("Doctor, consider Pneumonia in the right lower zone (moderate confidence); the note states "fever".") and leaves the phrase out only when no region was named. A cough no longer supports a mass, nodule or lung lesion (only weight loss and haemoptysis do), and weight no longer matches weight loss for a fracture (only bearing weight does). Recommendation to P4 (status rule), from the live runs: a finding whose image evidence failed the faithfulness test (unfaithful) can still become verified on one non-specific symptom, and the report then states it at the model's tier. On the brain sample, glioma with unfaithful and a note that only says headaches is reported as consider glioma (high confidence). Plan D1 says a region is verified only when the deletion drop clears its threshold, so I suggest compute_status returns uncertain when every image evidence item has faithful is False and the supporting text is the only evidence, and that the tier shown in the report is capped at moderate in that case. That is P4's function, so I have not changed it; the firewall already blocks claims for rejected findings and requires wording to match the tier. For P4 and P1 (wiring): pipeline._reader_step needs the brain, skin and bone readers registered, otherwise those studies stop at the reader. They load with readers.classifier.ImageClassifierReader.from_dir(ml/artifacts/<skin_cls|brain_cls>, modality=...) and readers.bone.BoneReader.from_dir(ml/artifacts/bone_det), and their run(ctx, reader=...) stage functions already leave ctx.findings for later stages. I tested my stages on all three that way. The precedents stage loads MedSigLIP on its first call (10-35 s on a CPU), so warm it up at server start when you add the model warm-up. For P4 (ready now, see Contract change request P3-1): from medproof.reasoning_stages import stage_specs, merge_findings, collect_claims. stage_specs() returns four StageSpecs for PIPELINE: second_read (MedGemma through MEDGEMMA_URL, or the seed in demo/medgemma_reads), context (injection guard, extraction, contradictions; reads ctx.notes), report (claims through the firewall and entailment judge; returns payload["claims"] and never findings) and precedents (MedSigLIP index for skin and bone). Each is cacheable=False because the notes change the result and the cache key is the image hash. Each never raises and degrades with a warning when its service is missing. The status rule needs no change: concordance.apply leaves second_read.agrees and box_iou empty for unvalidated modalities, so a second-reader disagreement is an audit flag (second_reader_disagrees) and never makes a finding discordant; the UI chain step for the second read will therefore show as inconclusive, not failed, on those modalities. For P4 (stages ready to wire, in this order): context.voice.process_dictation (optional audio) -> context.injection_guard.InjectionGuard.check -> context.extract.extract -> context.contradictions.check -> report.drafts.draft_claims (includes the firewall) -> report.entailment.judge -> report.fhir.build_bundle. A report consumer must show only claims with blocked_reason is None; a claim removed by the judge keeps its rendered text so the audit view can strike it through. build_bundle writes only reportable claims and redacts note quotes unless include_quotes=True. For P4: report.drafts.draft_claims(view, pool) returns claims + a StageResult(stage="report"); with pool=None or Groq down it returns the template-only report and says so in warnings. context.contradictions.check(...) returns findings with TextEvidence attached (one te_n id per finding/fact pair) plus missing_context prompts. The status rule can count polarity == "supports" text evidence.

P4 — Experience & Platform

Now: P4.1 to P4.7 done. P4.3 added the orchestrator (pipeline.run_study, core/{context,config,cache}.py) — see the shared log for that pass's detail. P4.4+P4.5 add:

  • core/ledger.py (Ledger): one global, append-only, hash-chained JSONL (not per-study — matches the single GET /ledger/verify endpoint). append() hashes prev_hash + event + payload_hash + ts + actor + study_id, so tampering any of those fields alone (not just the payload) is caught. verify() replays the whole chain.
  • core/store.py (StudyStore): in-memory, per-study bookkeeping for the API — status, accumulated StageResults, final StudyResult, an asyncio.Queue for live SSE subscribers. Not persisted: a server restart loses every in-flight/completed study. That's a known, stated limitation, not an oversight — plan.md's P4.4 Done-when doesn't ask for persistence, and the ledger is the durable record.
  • api/{app,studies,system}.py (FastAPI + sse-starlette): POST /studies (multipart upload, runs the pipeline on an executor thread, returns 202 immediately), GET /studies/{id}/events (SSE: one stage event per StageResult, a final done with the full StudyResult), GET /studies/{id}, POST /studies/{id}/feedback (the only place a doctor's accept/reject reaches the ledger — D17), GET /studies/{id}/fhir / GET /metrics / GET /model-cards (honest "not available yet" bodies — P3.13/P2.14/P2.13 aren't built, so these never fabricate data), GET /ledger/verify.
  • pipeline.run_study gained one additive parameter, on_stage_result (callback fired after every stage, cache hit or miss) — the API uses it to push SSE events and append ledger entries; nothing in pipeline.py itself imports the ledger, keeping it usable standalone. The API overwrites StudyResult.ledger_head with the real ledger's head after a run; direct run_study() callers (tests, future batch tooling) still get _fold_ledger_head's in-memory provisional value, now clearly the "no ledger wired" fallback rather than the only option.
  • Verified end to end with a real running server (not just TestClient): started uvicorn medproof.api.app:app, uploaded a real phantom PNG, confirmed GET /studies/{id}, POST /studies/{id}/feedback, and GET /ledger/verify all behave correctly over real HTTP (ledger entries went 2 → 3 after feedback, ok: true throughout).

Honesty note carried over from P4.3, now doubly true: the generalist/MedGemma stage still doesn't exist (P3.1 todo), so "kill MedGemma mid-run" is still unverified as literally stated — proven instead via the same stand-ins (synthetic stages, real brain/skin/bone degradation), now exercised through the HTTP API too, not just run_study() directly. P4.6 + P4.7 (UI), added the same day:

  • Landing / ("Two lines of sight"): ten chapters, each a real verification stage, around one persistent WebGL2 radiograph. Prerendered HTML (readable with JS off), hydrated, then the story engine (GSAP 3.15 ScrollTrigger + SplitText, Lenis, own GL renderer) loads after load/idle. Phones, reduced motion and devices without WebGL2 get an article layout with the same copy.
  • Workstation /read: study rail (four sample cases, upload), WebGL viewport, findings panel, verification chain, notes with two-way span linking, action bar, command palette, shortcut sheet. Live mode = POST /studies + SSE; sample mode = static web/public/cases/* (works without the backend).
  • Pages: /validation (real reports/leakage.json; "publishes at gate G3" where P2 data is pending), /models (declared resources with licences; /model-cards when P2.13 lands), /audit/:id (server verify + in-browser entry-hash recompute), /report/:id (A4 print), 404.
  • Backend additions (additive): GET /studies/{id}/image|pixels|artifacts/{name}|ledger; refs rewritten to URLs before storing; artifacts content-addressed by input sha256; intake no longer cached and failed stages never cached (both were real bugs found while driving the UI live; regression tests added).
  • Media is all CC0 / CC BY / public domain and listed in web/public/media/credits.json: Häggström chest PA (CC0), FracAtlas wrist (CC BY 4.0), ISIC_0015552 + Task 1 mask (CC-0), AFIP glioblastoma MRI (public domain). scripts/bake_web_assets.py ran the real CXR reader, anatomy segmenter, deletion test, eight perturbations and quality gate on the chest film; landing numbers come from that bake.
  • Measured: critical landing JS 83.9 KB gz (React 69.3 + landing 10.3 + shared 4.3); story engine 64.7 KB gz, deferred; CSS 11.1 KB gz. Landing HTML 22.4 KB prerendered.

P3-1 applied (2026-10-07): StudyContext.stage_results (appended right after each stage, cache hits included); findings merge one per id with the latest stage winning, then compute_status; StudyResult.claims collected from stage payloads; P3's four stages registered after the reader; GET /studies/{id}/fhir returns P3's validated R4B bundle for a finished study (409 while running). Workstation: an empty second_read.agrees shows as "Inconclusive" (never "agrees"), and the Firewall step counts passed and blocked claims; the report page lists only unblocked claims and downloads the FHIR bundle. Live run on the CC0 chest film: six stages, five findings (not 20), P3's demo second read merged onto each; context and report degrade here because this machine has no Groq key, so note evidence and claims were not exercised live. P1 stages wired (2026-10-07, P1's recipe): intake → router → reader → faithfulness → stability → P3 stages. The router runs only when no modality was given and is trusted at confidence ≥ 0.60 (RouterConfig.trust_confidence); its ood/router_uncertain flags are copied onto every finding. The reader dispatches chest (readers.cxr), brain and skin (readers.classifier, bundles brain_cls, skin_cls) and bone (readers.bone, bone_det) from PipelineConfig.models_root (MEDPROOF_MODELS_DIR, default ml/artifacts); missing weights degrade the reader with P1's reason. Faithfulness runs on the MEDPROOF_FAITHFULNESS_TOP_K (default 3) most probable findings; the rest stay untested and are withheld. Live chest run on the CC0 film: 51 s end to end (reader 6.6 s warm-ish, faithfulness 35 s for three findings, stability 2.6 s); Lung Opacity, Pneumonia and Consolidation came back verified (faithful, flip rate 0), Infiltration and Mass withheld as untested. Brain, skin and bone are wired and unit-tested with stubs but not run live here: their weights are not on this machine. UI audit and calibration (2026-10-07): a calibration stage (after stability, before P3's report) applies P2's calibrators from models_root: temperature + APS conformal set + tier for brain and skin, Platt + abstention band for bone; chest stays uncalibrated with a stated reason (product weights saw RSNA; P2's chest calibrator belongs to the chex weights). Live: glioma 0.86→0.95 {glioma}, nevus 0.87→0.90 {nv}, fracture 0.61→0.89 (high tier, so it is now verified on the note's support). The workstation now shows what the backend already produced but the UI dropped: quarantined instructions in the notes (boxed, struck through, explained), the context stage's missing-history prompts, finding flags in plain words, image-quality reasons with their fixes; the selected finding now owns its note spans. Validation shows P2's six figures; Models shows the datasheets. Not in the UI because nothing produces them on this machine: precedents (needs P3's retrieval index and MedSigLIP), router and OOD score (MedSigLIP), masks (brain segmenter and MedSAM not wired), voice dictation (P3 module has no API route yet). For the team (status rule): a finding is verified when its image region passes the deletion test or a note supports it. On the live demo cases the brain glioma and the bone fracture are verified by the note alone (glioma's region failed the test; bone boxes cannot take it). The UI shows the failed region in red, but a clinician may still challenge "verified". Decide whether a failed or untestable region should cap a note-supported finding at uncertain (a one-line change in core/status_rule.py plus tests). Assigned remaining work (2026-10-07, demo laptop = P4's). P4 wires each item into the pipeline, API and UI as soon as it lands; please reply in the shared log.

  • P1-A · Masks (brain U-Net, MedSAM). readers.classifier.run and readers.bone.run go straight from predict to to_findings, so no mask ever reaches a finding. Add optional unet= / medsam= arguments that call readers.segmenter.attach_unet_masks / attach_medsam_masks before to_findings, writing the mask PNG under ctx.artifact_dir. Tell P4 the loader calls (e.g. UNetSegmenter.from_dir(ml/artifacts/brain_seg), the MedSAM constructor) and P4 passes them from the pipeline. On the demo laptop: MedSAM weights as a local folder (MEDPROOF_MEDSAM_MODEL) and segmentation-models-pytorch. Done when a live brain study and a live skin study show the Mask layer.
  • P1-B · Faithfulness for box-only findings. Bone findings have no heatmap, so verify.faithfulness returns "not assessable" and bone can only ever be supported by notes. A box-region deletion test (blur the box against random boxes of equal area, same permutation rule) would let the image count. Done when bone findings get faithful true or false.
  • P1-D · Router and OOD on the demo laptop (moved from P3: MedSigLIP and the probe are on P1's machine). The router stage is wired but off here: it needs MedSigLIP (google/medsiglip-448, as a local folder via MEDPROOF_ROUTER_MODEL) plus router_probe.npz and ood.npz (sha256 in P1's section) under backend/artifacts/. Done when an upload without a chosen modality is routed and ood flags appear.
  • P1-E · MedSigLIP for precedents. P3's precedents stage embeds with the same MedSigLIP model; the local folder from P1-D covers it. Please hand it over together with P1-D.
  • P1-C · FYI. On the demo brain image (AFIP glioblastoma) the classifier's heatmap sits on the lower-left skull edge while the tumour is on the right; the deletion test correctly fails it. Worth a look before the panel.
  • P3-A · Precedents on the demo laptop. Needs the indices under ml/artifacts/retrieval/ (skin, bone, chest); the MedSigLIP model comes from P1 (P1-E). Licence constraint (only CC-0/CC-BY images in the web app): FracAtlas thumbnails (CC BY 4.0) may be shown; HAM10000 (CC BY-NC) and RSNA (competition rules) may not, so those precedents show label and similarity only. Please mark which datasets' thumbnails are displayable; P4 adds the thumbnail route and a "similar confirmed cases" panel. Done when a live skin or bone study returns precedents.
  • P3-C · Voice dictation. context.voice.process_dictation has no API route. Confirm it is final and that GROQ_KEY_AUDIO works, and say whether the transcript replaces or appends to typed notes; P4 adds an audio field to POST /studies and a record button. Done when a dictated note shows its spans in the workstation.
  • P3-D · Decide the note-only "verified" policy (see "For the team" above); P4 makes the one-line status-rule change and tests once decided.
  • P3-E · Report wording. Live claims omit the region ("Doctor, consider Pneumonia (moderate confidence)…" with no "in the right upper zone"), and "productive cough" is counted as supporting Mass. Deliveries checked (2026-10-07): P1-A masks, P1-B box faithfulness, P1-C off-target flag, P1-D/E router hand-over script, P3-A licence-gated thumbnails, P3-C dictation, P3-D decision (applied as P3-2), P3-E wording are all on main and verified live on this laptop except where a file is missing here: masks for brain (U-Net), skin and bone (MedSAM) are served and drawn; bone is verified by its own box (drop 0.58); chest sentences name the region. P4 added POST /transcribe and a Dictate button, a "similar confirmed cases" panel, the P3-2 status cap, and a cache-key fix (verify caches keyed on their input findings; a stale cache had silently dropped the new masks). Still missing on this laptop, by hand: MedSigLIP (the Hugging Face account behind .env's token has not accepted its terms, so fetch_models.py was refused: copy P1's backend/models/medsiglip), P1's backend/artifacts/router_probe.npz and ood.npz, and P3's retrieval_pack.zip (unzip to ml/artifacts/retrieval/). MedSAM was fetched (backend/models/medsam); start.bat now points the app at both model folders when present. Note: P3's demo/DEMO_GUIDE.md assumes P3's laptop (with MedGemma on its GPU) as the demo machine; the team should pick one. Known gaps (P4-owned): (0) a cold first chest read is ~17 s (reader timeout is now 45 s); a startup warm-up is still the fix for the first-stage target. (1) live chest findings outside the faithfulness top 3 are withheld as untested; raising MEDPROOF_FAITHFULNESS_TOP_K trades ~12 s per finding on CPU. Serving-time calibration is now wired (see below); the brain segmenter mask and MedSAM masks are still not wired, so the Mask layer stays empty. (2) First SSE stage: ~720 ms cold, ~180 ms warm, against the 500 ms target; needs a startup model warm-up. (3) Viewer 60 fps on 2048² is untested on a real GPU; this machine's headless runs use SwiftShader, which can't measure it meaningfully. (4) Lighthouse not run yet. (5) View Transition handoff landing → workstation not verified in a supporting browser. Run it: start.bat (repo root) installs missing deps, starts the API on :8000 (loading keys from the root .env via uvicorn --env-file) and the site on :5173 in their own windows, opens the browser, then runs scripts/warm_demo.py in the background so the four sample images hit the cached checks during a demo (chest: ~50 s cold, ~8 s warm). start.bat preview serves the production build on :4173. Next: P4.8 (Docker + CI wiring for the existing pytest/vitest/Playwright suites), startup warm-up for the 500 ms first-stage target, then P4.9 demo pack and P4.10 README. Blockers: none. Known gaps to flag, not blockers: in-memory StudyStore isn't persisted (above); the SSE queue fans out to at most one live drainer per study (fine for today's single-viewer-per-study UI, would need rework for multi-viewer); no make/pnpm binary on this machine so make setup/make dev are untested end-to-end here (the pieces run fine individually — uvicorn was started and curled directly); ml/ still has no dependency declarations of its own.

Decisions

Deviations from plan.md, newest last. Format: date · role · decision · evidence.

  • 2026-10-06 · P1 · Pulled verify/perturbations.py forward from P1.11 · shared by the quality-gate fixtures, the stability score and P2.12, so it is defined once.

  • 2026-10-06 · P1 · Added intake/preprocess.py (not in plan) · one preprocessing path for training notebooks and inference prevents train/serve skew.

  • 2026-10-06 · P1 · Blur metric is Laplacian variance divided by intensity variance, computed on a 512 px copy · raw Laplacian variance changes with contrast and resolution; test shows scores within 35% across a 512 to 1024 px resize.

  • 2026-10-06 · P1 · CXR tilt is estimated from the rotation that maximises left-right mirror symmetry (with an off-centre shift allowed), not the lung-mask axis · the anatomy segmenter arrives with P1.6; replace then. Only a tilt with symmetry gain of at least 0.03 is reported.

  • 2026-10-06 · P1 · PHI scrub keeps PatientAge and PatientSex, blanks names and identifiers, removes free-text descriptions, shifts dates by one random per-study offset that is not recorded, remaps UIDs, drops private tags, and warns on burned-in annotation · the context checks need age and sex; this is a PS3.15 subset, not a certified de-identifier.

  • 2026-10-06 · P1 · Quality thresholds are provisional values tuned on a synthetic phantom: the clean phantom passes with zero reasons across 8 seeds, and 100% of noise, blur and contrast degradations at severity 4 and 5 fail the gate · must be refit on real validation images.

  • 2026-10-07 · P1 · Wrote the Grad-CAM family (gradcam, gradcam++, hirescam) in readers/cam.py instead of using pytorch-grad-cam · fewer heavy dependencies on a slow network, full control for the saliency sanity check (P1.10), and unit tests against a model with a known answer. Default is gradcam++, configurable (MEDPROOF_CXR_CAM); P1.10 picks the final variant.

  • 2026-10-07 · P1 · Reader findings leave as status="uncertain", prob_calibrated=prob_raw, conformal_set=[], flag uncalibrated · a reader cannot verify itself; faithfulness (P1.9), stability, second read and the status rule upgrade it, and calibration fills the rest. No contract change.

  • 2026-10-07 · P1 · One ImageEvidence (kind heatmap, with bbox_xyxy, heatmap_ref, region_name) per finding; a finding with neither a box nor a heatmap is withheld with a warning, and a flat CAM is flagged no_localization · a finding with no image location fails the judges' rule.

  • 2026-10-07 · P1 · Changed cxr_xrv preprocessing from stretch to centre-crop (see Verified facts) and added cxr_anatomy (512 px, same crop). Only P1 code and tests used cxr_xrv; P2 is told in the handoff above.

  • 2026-10-07 · P1 · Positive threshold 0.5 (each label's own operating point) and tiers high >= 0.8, moderate >= 0.65, else low · provisional until P2 calibration; they live in readers/cxr_config.py. Labels are TorchXRayVision's names until P4 supplies core/vocab.py.

  • 2026-10-07 · P1 · Anatomy pixels inside the heart mask count as heart when naming ("cardiac region"); lung thirds still use each full lung extent · found on the real sample film, where Cardiomegaly was first named "left middle zone".

  • 2026-10-07 · P1 · Reader degrades instead of failing: no segmenter gives findings without region names and flag anatomy_unavailable; a heart on the image-left gives flag laterality_uncertain; model load failure gives a failed StageResult.

  • 2026-10-06 · P1 · Python 3.12 venv with pip instead of 3.11 with uv (uv not installed here); backend/pyproject.toml is minimal for the platform owner to absorb · no make smoke exists yet, so verification is python -m pytest in backend/.

  • 2026-10-06 · P1 · core/schemas.py is a verbatim copy of the section 4 sketch because the contract was not yet committed · platform owner replaces it; P1 code only uses StageResult.

  • 2026-10-06 · P2 · FracAtlas source corrected: figshare file 65518038 (v7, CC BY 4.0, md5 fe9da2c7...; dataset.csv lists 4,083 images, 719 fractured, the figshare text says 4,073), resolved at run time through the figshare API · plan.md's file id 43283628 returns HTTP 202 with no body.

  • 2026-10-06 · P2 · HAM10000 metadata is fetched from Harvard Dataverse (doi:10.7910/DVN/DBW86T, no login); Kaggle is the fallback only · removes the Kaggle-token dependency for metadata.

  • 2026-10-06 · P2 · Official ISIC 2018 Task 3 test metadata (age, sex, site, lesion_id) is fetched from the ISIC API · lets the official test set carry lesion-grouped CIs and a subgroup audit.

  • 2026-10-06 · P2 · Splits get a separate conformal calibration set: train 70 / val 10 / cal 10 / internal test 10 (HAM10000, brain), 65/10/10/15 (FracAtlas), LGG fold 0 of a 5-fold patient CV · temperature scaling uses val, conformal uses cal, so neither sees test.

  • 2026-10-06 · P2 · Own 64-bit DCT pHash in ml/data/phash.py instead of imagehash; duplicate groups = native id (lesion, patient) plus Hamming <= 4 edges; official test sets are never modified · avoids a PyWavelets build on new Pythons and guarantees a duplicate cannot straddle splits.

  • 2026-10-06 · P2 · Brain classifier is trained twice (original Kaggle split vs leakage-free split) · measures the inflation gap instead of estimating it.

  • 2026-10-06 · P2 · Bootstrap: percentile cluster bootstrap, B=2000, class-stratified for balanced accuracy, BCa for per-patient Dice; bone sensitivity at 90% specificity uses a threshold fixed on val · plan said B=1000.

  • 2026-10-06 · P2 · YOLO26n (released 2026-01-14) for the bone detector, yolo11n as fallback, via config · closes the [VERIFY] on the latest nano model.

  • 2026-10-06 · P2 · .gitignore carve-out !ml/data/ plus ml/data/raw/ ignored (the data/ rule hid ml/data/); ml/Makefile.inc supplied for P4 to include · P4 owns both files, so P4 please ack.

  • 2026-10-06 · P2 · gen_notes.py (synthetic notes) is left to P3; P2 supplies seed columns (age, sex, site, body part, label) in ml/data/splits/*.csv · P3.4 owns note content and injection cases.

  • 2026-10-06 · P2 · Flag for P1/P4: the LGG segmenter is trained on 3-channel TCGA data; the brain classifier data is single-channel T1-CE. Masks on classifier-style uploads are out of domain, and the model card will say so. Channel-replication augmentation reduces the gap but does not remove it.

  • 2026-10-07 · P2 · BDNeuro-MRI is not an independent external test: 4,238 of its 5,941 images (71%) are near-duplicates (pHash Hamming <= 4, pixel correlation median 0.93 to 1.0 against 0.47 for random same-class pairs) of images in the Kaggle brain dataset. The external test uses only the 1,644 images with no match (ml/data/splits/bdneuro.csv, split == external_test) · its README claims hospital provenance and exact/near-duplicate removal only within itself.

  • 2026-10-07 · P4 · Acked P2's .gitignore carve-out and ml/Makefile.inc (2026-10-06 entry above) · both already correct; root Makefile now includes ml/Makefile.inc unchanged.

  • 2026-10-07 · P4 · G0 contract freeze (P4.2) is purely additive: every existing field/class in schemas.py is untouched, confirmed by grepping all of backend/ for core.schemas imports (only Finding, ImageEvidence, StageResult are consumed anywhere today) · no Contract change request needed; nothing P1/P2 built against it changes shape.

  • 2026-10-07 · P4 · StudyContext (plan.md section 4's stage interface) is deliberately not defined as a class yet; intake/run.py already works via duck-typed ctx.raw_bytes/ctx.modality_hint/ctx.path · belongs with the P4.3 orchestrator, where stage ordering and caching actually need a concrete type.

  • 2026-10-07 · P4 · Pydantic's auto-generated per-field "title" is stripped before writing contracts/*.schema.json and before the TS generation step · untitled, each field became its own hoisted top-level TS type (e.g. Label1) instead of an inline property; titles aren't part of the contract, field names/types are.

  • 2026-10-07 · P4 · StudyContext (core/context.py) is a plain dataclass, not pydantic · it carries live DecodedImage/numpy data that has no business on a wire-contract model; kept duck-type-compatible with what P1's existing getattr(ctx, name, default) stages already read.

  • 2026-10-07 · P4 · The orchestrator's intake step calls intake.run_bytes() directly instead of the generic intake.run(ctx) stage wrapper · run(ctx) discards the DecodedImage it decodes, but the reader stage needs it; run_bytes's own docstring says it returns both specifically so a caller can do this. No change to P1's intake module.

  • 2026-10-07 · P4 · Added diskcache to backend/pyproject.toml dependencies (not dev) for core/cache.py · already plan-approved in plan.md section 5.5 (Apache-2.0), logging here for visibility rather than because it needed a CCR.

  • 2026-10-07 · P4 · Per-stage timeout uses a daemon-less ThreadPoolExecutor with future.result(timeout=...), shut down with wait=False on timeout · signal.alarm isn't available on Windows (the team develops on Windows laptops and Linux Kaggle/Colab alike); multiprocessing was rejected as pickling DecodedImage/reloading a torch model per call costs more than the problem it solves. Caveat that matters: this stops the orchestrator from waiting, it cannot forcibly kill a CPU-bound native stage (e.g. a blocking torch forward pass) — the thread keeps running in the background and its result is discarded. Correct for stages that raise or block on I/O (HTTP calls to MedGemma/Groq); not a hard kill switch for a hung CPU-bound stage.

  • 2026-10-07 · P4 · StudyResult.ledger_head is filled by folding sha256(prev + stage + payload_hash) over the stage results in-memory (pipeline._fold_ledger_head) · explicitly provisional, not the real ledger: no persistence, no append-only JSONL, no /ledger/verify. That's P4.5. The fold formula is meant to generalize into the real one, not be thrown away.

  • 2026-10-07 · P4 · P4.3's stated Done-when ("kill MedGemma mid-run") is not literally tested: the MedGemma/generalist stage doesn't exist yet (P3.1 todo) · verified instead with a synthetic slow/raising stage (mechanism) and the real brain/skin/bone "no reader yet" case (an actual present-day degradation). Flagging this rather than overclaiming the Done-when was met as written.

  • 2026-10-07 · P4 · The evidence ledger (P4.5) is one global JSONL, not one per study · matches plan.md's single GET /ledger/verify route (no study id in the path); entries carry study_id so a future per-study audit view (P4.6) can filter client-side without a new endpoint.

  • 2026-10-07 · P4 · StudyStore (API-side, in-memory, P4.4) is explicitly not persisted — a server restart loses every study · plan.md's P4.4 Done-when doesn't ask for persistence, and the ledger is the durable record; revisit only if a demo scenario needs studies to survive a restart.

  • 2026-10-07 · P4 · Ledger/store are constructor-injected onto app.state in api/app.py, not module-level singletons like readers/cxr.py's get_reader() · the API specifically needs per-test isolation since many pytest tests share one process; a global would leak state between them. cxr.py's singleton pattern is for an expensive model load, a different concern.

  • 2026-10-07 · P4 · GET /studies/{id}/fhir, GET /metrics, GET /model-cards return an honest {"available": false, ...}/empty body rather than fabricated data, since P3.13/P2.14/P2.13 don't exist yet · consistent with the no-fabrication stance already applied to pipeline.py's placeholders (ood_score=0.0, claims=[]).

  • 2026-10-07 · P4 · The per-study SSE queue fans out to at most one live drainer at a time (a second concurrent subscriber only gets the terminal state once the first drains it) · matches today's single-viewer-per-study UI; true multi-viewer broadcast would need a pub/sub fan-out, not asked for yet.

  • 2026-10-07 · P4 · Landing expands plan §10.5's "boldness in one place" into a full scrollytelling narrative at the owner's request; every chapter is a real verification stage with a provenance caption (real output, dataset annotation, or illustrative), and the app stays quiet · owner request for an awwwards-level landing.

  • 2026-10-07 · P4 · GSAP 3.15 (ScrollTrigger, SplitText) under its Standard no-charge licence, approved by the P4 owner; Motion, DrawSVG and twgl.js dropped (a small own WebGL2 helper shared by story and viewer) · one animation engine per tree; fewer dependencies.

  • 2026-10-07 · P4 · Two Vite entries: index.html (landing, no router, prerendered with renderToString then hydrated) and app.html (React Router lazy routes); a Vite plugin rewrites /read|validation|models|audit|report to app.html in dev/preview, P4.8's server must do the same · keeps the landing light and readable without JS.

  • 2026-10-07 · P4 · Scroll story uses sticky stages and treats the scene as a pure function of scroll position, not ScrollTrigger pins or a master timeline · React-safe (no pin-spacers), identical layout to the prerendered HTML, End key/anchor jumps can't leave stale state.

  • 2026-10-07 · P4 · Phones (< 1024 px), reduced motion and no-WebGL2 get an article layout (same copy, still images), predicted in an inline head script before paint · no layout jump; story engine never downloads there.

  • 2026-10-07 · P4 · Brain sample is a public-domain AFIP glioblastoma MRI (Wikimedia), not the Kaggle repackage or figshare · licence verifiable on its own page; brain mask layer stays empty until P2.4.

  • 2026-10-07 · P4 · Wax red gets a text shade --color-pencil-red-ink: #e8695e; strokes keep plan §10.2's #e2574c · #e2574c text measures 4.48:1 on film-base and 3.97:1 on panel (below WCAG AA 4.5:1); found by the axe suite.

  • 2026-10-07 · P4 · Browser ledger verification recomputes the entry-hash chain (string-only material) for real entries; payload re-hashing in JS uses Python-compatible canonical JSON and refuses floats (Python prints 1.0, JS 1), so float payloads are verified server-side · parity test pins JS output to a Python-generated fixture.

  • 2026-10-07 · P4 · Pipeline caches are keyed by content: artifacts live under artifact_root/<input sha256>, intake is uncacheable (it must set the decoded image), failed stages are never cached; reader cache_version bumped to v3 · three real bugs found driving the UI against a live server.

  • 2026-10-07 · P4 · Stage cache keys include a digest of the findings the stage starts from · cached faithfulness/stability results carry full finding copies; after P1 added masks to the reader, a cache hit from before won the merge and silently dropped every mask. Regression test test_a_changed_reader_output_invalidates_cached_verification.

  • 2026-10-07 · P4 · Applied P3-2 (a note alone caps a finding with an image region at uncertain) and updated P3's test_support_stands_unless_every_image_evidence_failed cases for untestable regions from verified to uncertain to match · P3's own decision; the cases predate it.

  • 2026-10-07 · P4 · compute_status treats tier abstain as uncertain, like low · P2's calibrated tiers add abstain below low; without this an abstaining finding could be verified. Test in tests/test_calibration_stage.py.

  • 2026-10-07 · P4 · Chest findings are deliberately left uncalibrated at serving time · the product reads densenet121-res224-all, which saw RSNA, and P2's chest calibrators were fitted for the external chex weights (P2's handoff gives the two options; switching weights is a team decision, not made here).

  • 2026-10-07 · P4 · P1's stages wired with three deviations from the recipe: (a) the reader is no longer cached and faithfulness/stability are (v1), because P1's readers leave the live model on ctx and a cached reader would starve both verify stages; (b) the stage cache key now includes the modality, so the same bytes read as another modality never reuse a result; (c) faithfulness and stability read the latest merged findings, not the reader's ctx.findings snapshot, otherwise stability (the later stage) would overwrite the faithfulness result when findings merge · found while wiring; regression tests in tests/test_pipeline_wiring.py.

  • 2026-10-07 · P4 · P3-1 applied with two adjustments: the orchestrator keeps its own id-merge and claim collection (same semantics as reasoning_stages.merge_findings/collect_claims, pinned by a parity test), and P3's stages are registered behind an ImportError guard · importing reasoning_stages takes ~1.7 s and needs fhir.resources (backend[services]), so a hard import would stop the API from starting on an image-only install; start.bat now installs .[services].

  • 2026-10-07 · P2 · Brain classifier headline: the Kaggle benchmark inflation is measured inside one model. Model A scores 0.942 on the original Testing folder, 0.903 once the 727 images with a duplicate in Training are removed, and 0.865 once the 1,051 images with a same-scan neighbour are removed (n=549). The across-model difference A minus B (0.002, CI -0.020 to 0.026) shows no inflation and is confounded by different models and test sets, so it must not be quoted · B (leakage-free group split, test n=563) scores 0.940 [0.921, 0.958].

  • 2026-10-07 · P2 · Two Kaggle accounts: the first account's concurrent GPU limit of 2 stalled the brain jobs, so they ran on a second existing account (johannshonigeorge, owner agreed) · plan.md section 14 lists several accounts as the fallback; its token is kept in the gitignored .env.kaggle2.

  • 2026-10-06 · P3 · Slot frame enum on top of free templates, so the "Doctor, consider..." framing is structural · docs/report_slots.md

  • 2026-10-06 · P3 · Firewall rule R4 binds cited evidence to the finding named in a slot; adds a mandatory false-block test on good claims · docs/report_slots.md

  • 2026-10-06 · P3 · MedGemma service returns boxes on a normalized 0..1000 grid; pixel conversion happens in the reader (P3.2) · services/medgemma/README.md

  • 2026-10-06 · P3 · Groq pool adds RPM and daily counters, a total deadline, and PoolResult.source (live/cache/fallback) so degradation shows up in StageResult warnings · backend/tests/llm/test_groq_pool.py

  • 2026-10-06 · P3 · Pool defaults to json_object plus schema text in the system prompt, not strict json_schema; unsupported params (reasoning_effort, response_format) are dropped once on a 400 · live call succeeded without needing the drop

  • 2026-10-06 · P3 · Real-GPU smoke is a separate script (python -m services.medgemma.smoke); CI tests use a fake backend · services/medgemma/

  • 2026-10-06 · P3 · Added .cache/ and ml/artifacts/medgemma_reads/ to .gitignore (LLM cache can hold note text; cached reads are large) · plan.md 11.3

  • 2026-10-07 · P3 · MedGemma runs in bf16 compute, not fp16: fp16 returned an empty reply on the RTX 4060 (Gemma-family activations overflow). The loader picks bf16 when the GPU supports it, else fp32 · services/medgemma/loader.py, smoke log

  • 2026-10-07 · P3 · The second reader is asked for a short FINDINGS/IMPRESSION report with a research-evaluation framing, not strict JSON: the JSON-only prompt gave looping labels, refusals and reasoning-only output. Labels are derived deterministically from the text (negation and hedging aware) in verify/report_labels.py; boxes from MedGemma are not requested yet · services/medgemma/prompts, 44 label-mapping tests

  • 2026-10-07 · P3 · Injection guard is three layers (regex, Prompt Guard per sentence at 0.9, gpt-oss-20b classifier). On 28 fresh attacks: regex 4, Prompt Guard 4 (7 at 0.5), classifier 25, all three 27, with 1 false positive in 15 benign. The regex 15/15 on the corpus is in-sample and must not be quoted · reports/p3_guard_blind.json

  • 2026-10-07 · P3 · Gold quote conventions in gen_notes were changed after reading dev-split extraction errors (medication names without 'on', no compound symptoms, foreign negators kept, foreign age fact added). The test split had not been scored · commit "fix(notes): align gold quote conventions"

  • 2026-10-07 · P3 · Differential injection test measures (1) facts overlapping injected text and (2) F1 against gold on injected notes vs their clean twins, not identical fact sets, which only reflect run-to-run variance (5/15 identical) · ml/eval_p3/eval_context.py

  • 2026-10-07 · P3 · TextEvidence ids are unique per (finding, fact) pair so each id has one owner, as the firewall's evidence-owner rule (R4) assumes · context/contradictions.py

  • 2026-10-07 · P3 · Report claims are drafted as frames chosen by the model (ids only, no labels or quotes in its input), expanded by code into slot templates; offline fallback picks frames deterministically · report/drafts.py

  • 2026-10-07 · P3 · FHIR is validated against fhir.resources.R4B (the library ships R4B, R5 and STU3, no plain R4); the resources used are the same in R4 and R4B. The library checks structure and data types but not value-set codes, so status codes are kept as constants in report/fhir.py · backend/tests/report/test_fhir.py

  • 2026-10-07 · P3 · Entailment judge input shows the second reader as an explicit word ("agrees", "disagrees", "inconclusive", "unavailable"), a summary of what the notes contain (to verify "the notes do not mention history"), and readable flag phrases. These came from the first-pass errors on variant 0; variant 1 was written afterwards and is the fair measurement · reports/p3_entailment_*.json

  • 2026-10-07 · P3 · Voice: Whisper gets a static clinical-vocabulary prompt (no patient data) and the transcript is treated as an untrusted note: same guard and extraction as typed text · context/voice.py

  • 2026-10-07 · P3 · MedGemma service reads its prompt files once at start. A git rebase during a long batch removed a prompt file for a moment and killed the run; resumable by image id, so nothing was lost but time · services/medgemma/reader.py

  • 2026-10-07 · P3 · The discordant flag is judged by two things: does it predict specialist errors (P2's signals.compare), and what share of the specialist's correct reports it would also demote. Bone passes the first and fails the second (83%); skin does not pass the first so far · ml/eval_p3/eval_second_reader.py, reports/concordance.json

  • 2026-10-07 · P3 · Two regex rules added to the injection guard after offline red-teaming (forget everything above; pretend or act as the attending), with tests that clinical uses of the same words are not flagged. The regex layer stays a cheap first filter, not the defence: recall on fresh attacks is carried by the classifier layer · backend/tests/context/test_guard.py

  • 2026-10-07 · P3 · The red-team suite is offline and deterministic (language models replaced by fakes) so it can run in CI and under make redteam; it covers P3 components only, the pixel-level cases are P1 and P4's · backend/tests/redteam, reports/redteam_p3.json

  • 2026-10-07 · P3 · MedGemma needs CUDA torch, which no existing venv has (.venv-gpu is the CPU build); a separate .venv-medgemma was created rather than changing .venv-gpu, and is excluded locally through .git/info/exclude · setup note, not committed

  • 2026-10-07 · P3 · P4's status rule demotes a finding whenever second_read.agrees is False or box_iou < 0.1, so the concordance stage now leaves both empty unless the modality is in DOWNGRADE_VALIDATED (empty). Disagreement stays visible as the flag second_reader_disagrees and in second_read.label · backend/medproof/verify/concordance.py, reports/concordance.json

  • 2026-10-07 · P3 · Finding labels are canonicalised before matching the second read (case, spaces, underscores, and the full skin names to the dataset codes), because P4's core/vocab.py names the skin classes in full while the classifier and the report parser use codes. Without it a finding called melanoma could never agree with a report saying melanoma · medproof/verify/report_labels.py::canonical_label

  • 2026-10-07 · P3 · The report lexicon takes its label terms from core/vocab.py plus local aliases, so a label added to the shared vocabulary is blocked outside a slot without editing the lexicon · tests/report/test_lexicon_vocab.py

  • 2026-10-07 · P3 · Pipeline stages are cacheable=False: the stage cache is keyed by (image hash, stage, version) and these stages depend on the notes and on live services · medproof/reasoning_stages.py

  • 2026-10-07 · P3 · Second reads for the four sample images are committed under demo/medgemma_reads (a new folder in P4's demo/, additive), keyed by the SHA-256 of the exact image.webp and the prompt version, so they only apply if that exact file is uploaded · demo/medgemma_reads, tests/test_reasoning_stages.py

  • 2026-10-07 · P3 · Third injection set (26 attacks, 16 benign, not used to develop the rules): regex 6/26 with 0 false positives, Prompt Guard at 0.9 9/26 with 1, the language-model classifier 25/26 with 1, all layers 26/26 with 2/16 benign flagged (both from model layers). The target of zero benign flagged is not met; the cost is withheld note text, not a missed attack · reports/p3_guard_blind_fresh2.json

  • 2026-10-07 · P2 · Conformal sets use the randomised APS with a zero conformity score for a true top class, and the set always contains the top class · the deterministic APS over-covered badly (coverage 0.985 at a 0.90 target on the brain set, 0.953 on skin), because wrong cases sit near score 1 and the quantile jumps the gap; forcing the top class into a randomised rule overshot too (0.958), so the score for a true top class is zero, which is valid split conformal and gives 0.898 on the official skin test. The per-input random draw is a hash of the logits, so a study always gets the same set.

  • 2026-10-07 · P2 · Binary readers (chest label, bone image score) use Platt scaling on logit(score) plus a conformal false-negative-rate threshold and an abstention band at calibrated probability 0.4 to 0.6, fitted on separate halves of the calibration data · temperature alone cannot fix the base rate of operating-point-scaled chest scores (ECE 0.220 to 0.018 with Platt); only Lung Opacity and Pneumonia can be calibrated on RSNA, the other 16 labels stay uncalibrated.

  • 2026-10-07 · P2 · make eval uses B=1000 bootstrap resamples (notebooks used 2000), B=500 for subgroup cells, B=300 for corruption cells and for the validation and calibration splits, and AUPRC and temperature-only ECE as point estimates · keeps the run near 100 s; intervals agree with the notebook ones to the third decimal on the headline metrics.

  • 2026-10-07 · P2 · Quality-gate flags and RSNA header demographics are committed (ml/artifacts/<model>/predictions/quality.csv, ml/data/splits/rsna_meta.csv) · both need the raw images, and make eval must run from a fresh clone.

  • 2026-10-07 · P2 · Brain similarity grouping uses Hamming radius 6, not 10, and LGG patients merge only on near-duplicate tumour slices · radius 10 chained 5,619 of 7,200 images into one group and dark top and bottom slices matched across LGG patients; both made the splits useless.

  • 2026-10-07 · P2 · The energy-score OOD baseline reaches AUROC 0.83 to 0.98 against other modalities and CIFAR-10, below the 0.95 target · P1.5 (MedSigLIP Mahalanobis) is still needed; the product decoder also rejects 59 truncated FracAtlas JPEGs (see For P1).

  • 2026-10-07 · P1 · Autonomous run: P1.4 to P1.12 were built without a fresh plan-mode approval per package, on the instruction to continue unattended; the approved roadmap plan covered the order and scope.

  • 2026-10-07 · P1 · Faithfulness pass rule is a permutation test, not a global threshold: a heatmap passes if its deletion drop at k=10% beats same-shaped random regions placed elsewhere (overlap at most 0.25) with p at most 0.10 and the drop is at least 0.05; n_random must be at least 9 or the config is rejected, because nothing could ever pass. Random regions pass at most 10% by construction; measured 0 of 18. Median fill is available beside blur because blur keeps silhouette evidence such as heart size (Cardiomegaly drop was negative under blur).

  • 2026-10-07 · P1 · Stability: severity 2, seed 0, a finding flips when it crosses its positive threshold in either direction; worst_perturbation is named only when something flipped; flag unstable above flip rate 0.25 (the plan's status-rule limit). Severity is configurable; P2.12 sweeps all five.

  • 2026-10-07 · P1 · Saliency sanity chooses among methods whose maps do not collapse to empty after randomisation (more than half the stages empty excludes a method). On the chest model gradcam and hirescam collapse once the classifier is randomised, so gradcam++ is chosen; the result is one image and should be repeated.

  • 2026-10-07 · P1 · Thread counts measured on the dev machine: chest DenseNet 0.33 s per image on 1 thread versus 4.3 s on 8 (default 1, MEDPROOF_THREADS); MedSigLIP 15 s on 1 thread versus 9 s on 6 (default 6, MEDPROOF_ROUTER_THREADS).

  • 2026-10-07 · P1 · Router uses a canonical model_id (default google/medsiglip-448, stored in the probe and cache keys) separate from an optional local model_path; a local folder used as the id overflowed Windows path limits and would have baked a machine path into the probe. Probe and OOD files are plain .npz (no pickle) and record the embedder id; a probe for another embedder is refused with a warning and routing falls back to zero-shot.

  • 2026-10-07 · P1 · Zero-shot fallback uses a softmax over cosine similarity times the model's logit scale across the five class prototypes (mean of several prompts per class, kept in router_config.py); SigLIP's own sigmoid and bias are not used because only the ranking across our classes matters, so its confidence is relative, not a calibrated probability.

  • 2026-10-07 · P1 · Router probe temperature fitted on the validation split reached its lower bound (0.05): the staged classes are almost perfectly separable, so the confidence is saturated and not informative. It needs harder, mixed data (P2's held-out mix, CT, fundus and pathology as other) before the confidence means anything. other is represented only by 60 CIFAR-10 natural images here.

  • 2026-10-07 · P1 · Router grouping: every image is its own group except brain MRI, where a patient folder is one group; FracAtlas patient ids were not used. Held-out evaluation counts only test groups.

  • 2026-10-07 · P1 · OOD threshold is the 95th percentile of cross-fitted (group k-fold) Mahalanobis distances over all non-test groups, with Ledoit-Wolf shrinkage; a plain 95th percentile of 6 to 15 validation points flagged up to a third of normal images. Energy score on the chest model's logits was implemented, calibrated and tested: AUROC 0.05 separating chest films from skin, brain, bone and natural images (the other images look more confident), so it is NOT used as a flag; the OOD flag is Mahalanobis only. Result in intake/results/ood_energy_cxr.json.

  • 2026-10-07 · P1 · MedSAM runs on CPU through transformers (flaviagiammarino/medsam-vit-base, Apache-2.0, from pytorch_model.bin); results cached by image hash, box and model id in memory and optionally on disk.

  • 2026-10-07 · P1 · Brain and skin readers are one generic ImageClassifierReader built from P2's model_meta.json (arch, classes, preproc_spec name or dict, weights hash); layer paths bn2 (EfficientNet) and stages.3 (ConvNeXt) were checked on the real timm models. Brain notumor is not reported as a finding. Bone image score is the highest box confidence, positive cut-off 0.25 until P2.8 calibrates it.

  • 2026-10-07 · P1 · Tests never download model weights: real-model tests run only when the weights are already on disk and otherwise skip with a reason; MEDPROOF_ROUTER=0 is set in tests/conftest.py so a locally trained router does not change test behaviour.

  • 2026-10-07 · P2 · SECURITY: a .env containing the P2 Kaggle API token was uploaded to the public repo through the GitHub web UI (commit 73b4105, 08:28) and untracked in d7c8f71, but it stays in git history · rotate that Kaggle token (kaggle.com > Settings > API > expire and create a new one) and the same file in any other role's keys if they were uploaded the same way; never upload .env, use the private channel. Removing the file from history needs a force-push that P4 must coordinate.

  • 2026-10-07 · P2 · Optional extras done. (1) MILK10k external skin test (4,566 dermoscopic images after removing 94 duplicates of HAM10000 and 576 images of classes the model does not cover): balanced accuracy 0.571 [0.543, 0.602] against 0.680 on the official test; conformal coverage 0.73 against a 0.90 target, so the guarantee fails under dataset shift. (2) Skin-tone audit on MILK10k (grades 0 very dark to 5 very light, 0 and 1 merged): no monotone trend, darkest groups only about 100 images. (3) Skin ensemble of three seeds with flip averaging: official test 0.728 [0.680, 0.775], MILK10k 0.609 [0.580, 0.636], +0.023 and +0.024 over the mean single seed (seeds differ by 0.04); recommendation ship it, registry entry skin_ens, calibrator ml/artifacts/skin_ens/calibration.json. (4) RAPS and Mondrian APS compared with the shipped randomised APS (reports/conformal_variants.json); building that comparison exposed a bug in the Mondrian set rule (top class not always named, coverage read 0.53), fixed in dfe9b05.

Contract change requests

Format: id · proposer · change · affected roles · P4 ack (yes/no) · applied in commit.

  • P3-2 · P3 · Status rule: a finding cannot be verified unless at least one of its image evidence items passed the faithfulness test. In core/status_rule.py, _is_uncertain gets one more line: if finding.image_evidence and not any(ie.faithful is True for ie in finding.image_evidence): return True, placed after the rejected and discordant checks. Today plan section 4 lets supporting note text alone verify a finding; with this, notes can lift a finding to uncertain but not to verified, which is what the judges' rule and plan D1 (a region counts only if deleting it matters) ask for. Effect on the demo files: bone, skin and brain findings that passed their test stay verified; chest findings outside the three that are tested become uncertain; no passing finding changes. Needs a test update in tests/test_status_rule.py for the case of a supporting note with an untested region. · P4 (status_rule.py), P1 (nothing) · P4 ack: yes, 2026-10-07 · applied in commit: see the P4 shared-log entry 'Deliveries checked and wired'

  • P3-1 · P3 · Let later stages see earlier results and let the study carry P3's output. (a) core/context.py: add stage_results: list[StageResult] = field(default_factory=list). (b) pipeline._run: ctx.stage_results.append(result) right after each stage, before on_stage_result. (c) pipeline._collect_findings: build the list with reasoning_stages.merge_findings(stage_results) (one finding per id, the latest stage wins) and keep compute_status on each; today every stage's findings are concatenated, so a stage that returns updated findings would duplicate them. (d) StudyResult(claims=collect_claims(stage_results), ...) instead of []. (e) PIPELINE.extend(stage_specs()); the specs carry their own timeouts (second_read 130 s because a live MedGemma read is 6-11 s and the client allows 120 s, context 25 s, report 45 s, precedents 30 s) and cacheable=False. (f) GET /studies/{id}/fhir: call report.fhir.build_bundle(findings, claims, study_id=..., issued=..., ledger_head=...) on the finished StudyResult; no stage is needed. Not a schema change: StageResult, Finding, Claim are untouched. · P4 (pipeline.py, core/context.py, api/studies.py) · Observed with the real run_study (reader step stubbed to set ctx.findings as P1's reader now does, then PIPELINE + stage_specs()): all four P3 stages run and return a StageResult, but the finished StudyResult.findings holds four copies of finding f1 (one per stage, each carrying only that stage's update: none has the second read and the note evidence together) and claims is empty. (c) and (d) fix exactly that; (a) and (b) matter for faithfulness and stability updates, because the reader's ctx.findings is only a pre-verification snapshot, which my stages fall back to when stage_results is absent. This also affects P1's faithfulness and stability stages, which return full finding lists too. · P4 ack: yes, 2026-10-07 (merge and claim collection kept in pipeline.py with a parity test against merge_findings/collect_claims; registration behind an ImportError guard; see Decisions) · applied in commit: 1a92ebd (orchestrator), 91b8459 (FHIR route), c38c217 (UI)

Verified facts log

Record anything marked [VERIFY] in plan.md once checked. Format: date · role · item · result · source.

  • 2026-10-07 · P1 · torchxrayvision 1.5.5 DenseNet densenet121-res224-all output · forward() applies sigmoid then operating-point scaling (op_norm) when op_threshs is set (it is, for every pretrained set), so outputs are probabilities with 0.5 at each label's operating point; labels with a NaN threshold (some chex and mimic sets) output a constant 0.5 · installed models.py source.

  • 2026-10-07 · P1 · Label set and order of -all · 18 labels, identical to xrv.datasets.default_pathologies; other weight sets blank some labels with empty strings · installed models.py and a real-weights test.

  • 2026-10-07 · P1 · Backbone layer for CAM · model.features is an nn.Sequential ending in norm5, applied before the final ReLU, so the captured map can be negative; Grad-CAM++ inverted the map on such inputs until activations were rectified (fixed, regression test) · installed models.py, test_cam.

  • 2026-10-07 · P1 · Input convention · single channel, range [-1024, 1024], square. The library's transform is XRayCenterCrop then XRayResizer(224); the PSPNet raises on non-square input · installed datasets.py and utils.py. Our prepare(..., "cxr_xrv") matches it to within a mean of 25 grey levels on a +-1024 scale and 0.08 in probability on a non-square phantom (resampling filter differs: cv2 INTER_AREA vs skimage).

  • 2026-10-07 · P1 · Anatomy model chestx_det.PSPNet · input square [-1024,1024] resized to 512, output raw logits (14, 512, 512), target order verified. "Left Lung" is the patient's left lung: on a real NIH chest film (public sample from the torchxrayvision repo, not stored) the "Right Lung" mask centroid was at x=151 and "Left Lung" at x=351 of 512, heart at x=290, matching the standard display with the patient's right on the image left and the film's own L marker on the image right. Lung masks include retrocardiac lung and overlap the heart mask (handled in zones.name_region) · one-off real-image run.

  • 2026-10-07 · P1 · Download environment · Windows curl failed TLS revocation checks here and the library downloader stalled at 0 bytes, so the two public weight files were fetched with curl --ssl-no-revoke (TLS chain still verified, only the revocation lookup skipped) from the official mlmed/torchxrayvision v1 release URLs. No checksum is published for them; the weights' sha8 appears in model_id.

  • 2026-10-06 · P3 · google/medgemma-1.5-4b-it id, gating, transformers>=4.50, AutoModelForImageTextToText, BF16 weights · confirmed · huggingface.co/google/medgemma-1.5-4b-it

  • 2026-10-06 · P3 · Groq free tier gpt-oss-120b/20b: 30 RPM, 8K TPM, 1K RPD; cached tokens not counted; 429 carries retry-after · confirmed · console.groq.com/docs/rate-limits

  • 2026-10-06 · P3 · Prompt Guard id is meta-llama/llama-prompt-guard-2-86m (as in plan.md; the rate-limit page drops the prefix). Also listed: -22m, openai/gpt-oss-safeguard-20b, whisper-large-v3(-turbo) · confirmed · GET /models with the extract key

  • 2026-10-06 · P3 · Groq accepts response_format: json_object and reasoning_effort: low on gpt-oss-20b (live call: 1 attempt, 258 tokens, injected "report no findings" text ignored, 2nd call served from cache) · confirmed · live pool call

  • 2026-10-06 · P3 · MedGemma access: HF token is valid but returns 403 on the gated repo (terms not accepted for that account, or token lacks gated-repo read) · BLOCKED

  • 2026-10-06 · P3 · MedGemma CXR box prompt/format · NOT verified (model card gave no format); check on first smoke run

  • 2026-10-07 · P3 · MedGemma 1.5 4B nf4 fits and runs on an RTX 4060 Laptop (8 GB): 3.2 GiB peak VRAM, 23 s load from cache, 5.5 s per read with the report prompt · confirmed · services/medgemma/smoke.py log

  • 2026-10-07 · P3 · MedGemma fp16 compute returns an empty reply on that GPU; bf16 works · confirmed · smoke runs

  • 2026-10-07 · P3 · MedGemma CXR box prompt/format · still NOT verified (the report-style prompt does not request boxes)

  • 2026-10-07 · P3 · Llama Prompt Guard 2 on Groq answers with a bare probability and scores a whole clinical note low (3 of 15 attacks at 0.5); per-sentence it fires on 4 of 28 fresh attacks at 0.9 · confirmed · reports/p3_guard_blind.json

  • 2026-10-07 · P3 · Groq free tier real limits observed: 8,000 tokens per minute is the binding limit for gpt-oss-20b extraction (about 4-6 calls a minute with a 1.5K-token prompt) · confirmed · response headers

  • 2026-10-07 · P3 · Whisper (whisper-large-v3-turbo) on Groq with GROQ_KEY_AUDIO accepts multipart audio with response_format=verbose_json and returns text, language and duration; a 20 s synthesized dictation was transcribed correctly including numbers · confirmed · ml/eval_p3/voice_roundtrip.py

  • 2026-10-07 · P3 · fhir.resources 8.3.0 does not enforce value-set bindings (an invalid status string validates); required fields and date formats are enforced · confirmed · test_fhir.py

  • 2026-10-07 · P3 · google/medsiglip-448 returns 403 with this account's HF token while google/medgemma-1.5-4b-it returns 200 · confirmed · HTTP check; accept the terms on the MedSigLIP page with this account to unblock

  • 2026-10-07 · P3 · The PyTorch CUDA wheel (cu128, about 2.9 GB) times out with pip's default read timeout on this network; --default-timeout 300 --retries 10 worked · confirmed · first attempt failed with ReadTimeoutError, second succeeded (torch 2.11.0+cu128, CUDA available)

  • 2026-10-07 · P3 · google/medsiglip-448 access now returns 200 for this account's token (terms accepted) and the embedder works with transformers 5 once it uses the image processor only and reads pooler_output from the output object · confirmed · retrieval builds on FracAtlas and HAM10000

  • 2026-10-07 · P3 · MedSigLIP vs BiomedCLIP precedents: FracAtlas precision@5 0.844 (0.823-0.865) vs 0.835; HAM10000 0.667 (0.643-0.692) vs 0.619, majority-vote balanced accuracy 0.486 (0.429-0.541) vs 0.362; MedSigLIP is kept as the retrieval embedder · confirmed · reports/retrieval__.json

  • 2026-10-07 · P3 · Brain MRI images downloaded with P2's downloader (needs pip install kaggle, KAGGLE_API_TOKEN, and PYTHONUTF8=1 on Windows: the downloader's subprocess reader crashes on cp1252 otherwise). The archive was 228,676,924 bytes against the pinned 164,670,110, so the downloader reported a size mismatch, but all 4,825 non-dropped split rows resolve to files and 60 of 60 sampled images have a perceptual hash identical to P2's stored one, so the images are the same and only the packaging differs · confirmed · ml/data/splits/brain_mri.csv

  • 2026-10-07 · P3 · Brain precedents with MedSigLIP: precision@5 0.918 (0.895-0.944) against 0.273 for chance, majority-vote accuracy 0.945 and balanced 0.952 (0.935-0.974), by class glioma 0.87, meningioma 0.92, no tumour 0.91, pituitary 0.98 · confirmed · reports/retrieval_brain_mri_medsiglip-448.json

  • 2026-10-07 · P3 · MedGemma chest boxes carry no localisation (100 RSNA test images with a true opacity, 50 normal). With a prompt that lets it decline it answers [] to all 150, so it finds 0 of 100 opacities. Told the opacity is there it boxes every image, but pointing-game hits are 6% (2-11%) against 10% (5-16%) for a fixed central box, IoU of at least 0.1 on 59% against 69%, mean best IoU 0.136 against 0.160, and all 50 normal films also get a box. The convention is [y_min, x_min, y_max, x_max] on a 0-1000 grid; the median forced box is [430, 350, 700, 650], the middle of the chest. Box overlap with the specialist stays out of concordance (box_iou is left empty by policy) · confirmed · reports/medgemma_boxes_rsna_optout.json, reports/medgemma_boxes_rsna_forced.json

  • 2026-10-07 · P3 · Chest precedents with MedSigLIP, index = RSNA calibration split (7,621 images), queries = the test eval batch (999): precision@5 0.666 (0.648-0.685) against 0.357 for chance, majority-vote accuracy 0.723 and balanced 0.716 (0.688-0.744); by class Normal 0.83, no-opacity-not-normal 0.60, opacity 0.57 · confirmed · reports/retrieval_rsna_medsiglip-448.json

  • 2026-10-07 · P3 · All four demo images through the default pipeline after P4 registered the readers and wired faithfulness and stability (real models, real Groq, shipped MedGemma reads, no service; first call per process includes the MedSigLIP load): chest 59.7 s (faithfulness 28.8 s, precedents 16 s), bone 15.3 s, skin 29.4 s, brain 62.6 s (precedents 45.8 s). Chest: 5 findings, 3 faithful and verified, 2 low-tier uncertain, 5 claims, 5 precedents each. Bone: faithfulness not assessable for a box detector, 1 finding, 1 claim. Skin: nevus fails the deletion test, rejected, no claim. Brain: glioma fails the deletion test (flag unfaithful) but is verified on the note's headaches, and the report says consider glioma, high confidence · confirmed · run on this machine

  • 2026-10-07 · P3 · A symptom no longer rescues a heatmap that failed the faithfulness test: contradictions.check marks a fitting symptom neutral when every image evidence item has faithful is False, so the finding stays rejected and no claim is made. This implements my earlier recommendation to P4 for the symptom case without touching compute_status. Effect on the live runs: brain findings that fail the test (most of them) are withheld instead of being reported at high confidence on the word headaches; a normal brain no longer produces a low-confidence glioma claim · backend/tests/context/test_contradictions.py

  • 2026-10-07 · P3 · Precedent thumbnails are licence-gated in code: THUMBNAIL_DATASETS = {fracatlas, brain_mri} (CC BY 4.0 and CC0); for ham10000 (CC BY-NC) and rsna (not redistributable) the stage never copies a thumbnail and thumb_ref is empty. Thumbnails are built ahead of time as 128 px PNGs (build_retrieval --thumbs-only, 2,447 bone and 3,349 brain, 52 MB), and my earlier thumbnail folders for ham10000 and rsna were deleted · docs/retrieval_indices.md, backend/tests/test_reasoning_stages.py

  • 2026-10-07 · P3 · Dictation appends to the typed note as one note and returns the merged text for the doctor to review; the audio never enters the study · backend/medproof/context/voice.py (append_dictation, dictation_to_note)

  • 2026-10-07 · P3 · Voice check, live: a synthesized 12 s dictation through Whisper on Groq (GROQ_KEY_AUDIO) came back accurately ("Fifty eight year old" as "58-year-old", "two thousand fifteen" as "2015") in 3.4 s; a second dictation containing a spoken attack was flagged by the guard and its sentence kept out of the facts. Neither the API (no audio field, no transcribe route) nor the website (no microphone or recording code) uses voice yet: the library works and is tested (34 voice tests), the wiring is P4's and is not done · confirmed · run on this machine

  • 2026-10-07 · P3 · Extraction now tolerates dropped filler words: a spoken or spelled-out note ("History of tuberculosis in 2015") lost its history fact because the model returned "tuberculosis 2015", which is not exact text in the note. A multi-word quote now also matches when only up to two small filler words (in, of, the, a, on, at, for, with, to, from, since, and) sit between its words; the fact's quote and span are taken from the note's own text, and a match never lands inside a flagged span. Held-out test split (62 notes, cached Groq answers), exact-span F1 0.919 to 0.922 (precision 0.922 to 0.906, recall 0.917 to 0.939); lenient F1 0.985 · backend/tests/context/test_extract.py, reports/p3_context_test.json

  • 2026-10-07 · P3 · P4's voice work verified live: POST /transcribe returns the merged note for real speech (also through the website proxy at /api/transcribe), rejects a wrong file type with a warning and an empty file with 422, and the Dictate button in Edge, driven with a WAV file as the fake microphone, recorded, transcribed and put the text in the notes box with the review message and 0 browser errors. Not tested: a person speaking in a noisy room · confirmed · backend/tests/api/test_transcribe.py, a Playwright run

  • 2026-10-07 · P3 · Startup warm-up (the cold-start gap P4 listed as known gap 0): api/warmup.py loads the four readers, the precedent embedder and the language-model pool in a background thread after the server starts, and wakes the MedGemma service with one tiny read (a unique image id each time so the reader's cache never answers). The server answers while it warms and a failed load never stops it. MEDPROOF_WARMUP chooses: 1 default, 0 off, or a list (cxr,second_read). Measured on a film the system had never seen, after a full restart: second read 67 s to 8 s, whole study 109 s to 50 s (what remains is the faithfulness test, about 23 s on a CPU, and the precedent lookup, about 9 s). The warm API holds about 4.4 GB, the same footprint it reached after a day of use, so on a memory-tight laptop warm only the chest path · backend/tests/api/test_warmup.py

  • 2026-10-07 · P3 · The sentence that cites the note keeps the region; a cough no longer supports a mass, nodule or lung lesion; weight no longer matches weight loss for a fracture · backend/medproof/report/drafts.py, context/contradictions.py

  • 2026-10-07 · P3 · Claims name findings in words and skip an unnamed region: display_label maps the dataset codes (bcc, nv, mel, pituitary) to names, and the plain frame is used when no region was named, so the report no longer reads consider bcc in the unspecified region · backend/medproof/report/render.py, drafts.py

  • 2026-10-07 · P3 · When a finding carries laterality_conflict, the report sentence is chosen by code, not by the model: the note and the image disagree about the side for the finding, please review the image directly, with no confidence stated. Without it a conflicted finding was still reported as consider X (high confidence) · backend/tests/report/test_drafts.py

  • 2026-10-07 · P3 · The automatic router (modality Not sure) does not work on this machine: it is off unless artifacts/router_probe.npz exists or MEDPROOF_ROUTER=1, and loading MedSigLIP's processor needs sentencepiece plus tokenizer files that the gated repo does not serve to this account (401 on spiece.model and special_tokens_map.json). Studies uploaded with Not sure stop at the reader; the demo always picks the modality by hand · confirmed · run on this machine; P1 to check the setup on their machine

  • 2026-10-07 · P3 · Demo pack and guide: demo/DEMO_GUIDE.md (run order for P1 to P4, start-up, troubleshooting, what not to claim) and a git-ignored demo_inputs/ of 16 files with paste-ready notes and warm_pack.py, which sends each file through the live API and prints what it should show; all 16 matched. The inputs are not a random sample: each modality has one proven and one withheld example, and some are chosen to show the quality gate and the laterality check · confirmed · demo_inputs/warm_pack.py

  • 2026-10-07 · P3 · RSNA download speed on this network swung between 0.2 and 2.5 MB/s (about 3.9 GB, roughly 1.5 hours); the downloader extracted all 26,684 training images; the DICOM decode costs 91 ms per image · measured

  • 2026-10-07 · P3 · P2's trained weights (bone_det, brain_cls, brain_seg, skin_cls) arrived as parallax_weights/; all four SHA-256 values match both the shipped list and the hashes in ml/artifacts/registry.json; copied to ml/artifacts/<model>/weights.pt (git-ignored by *.pt) and the staging folder is excluded locally · confirmed · sha256sum, registry

  • 2026-10-07 · P3 · P4's pipeline._reader_step registers only the chest reader, so brain, skin and bone studies end with no reader available for modality ... yet and no findings. With the real trained readers loaded by hand and then P4's run_study plus my four stages, on the demo images: bone 17.8 s, one located finding (fracture, low tier, uncertain), real precedents, one claim citing the note; skin 20.2 s, one finding (nv, high tier) which stays rejected because nothing supports it yet, so no claim; brain 35 s, one finding (glioma, high tier) verified on the note's headaches, one claim, precedents timed out at 30 s because the first call loads MedSigLIP on a CPU (10-35 s), so the stage timeout is now 90 s · confirmed · run on this machine

  • 2026-10-07 · P3 · A note that agrees with a finding on side is consistency, not support: contradictions.check now marks matching laterality neutral (a conflict is still contradicts), the firewall and the template report ignore laterality when deciding support, and the report cites a clinical quote (symptom, history, lab) or uses the plain frame. Found by running the real pipeline on the chest sample: the single word Right made all five findings supported and two of them verified before any faithfulness test ran, and every claim read the note states Right · backend/tests/context/test_contradictions.py, tests/report

  • 2026-10-07 · P3 · End-to-end run through P4's real pipeline (chest sample image, real chest model, real Groq, shipped MedGemma seed, no service): 12.7 s, intake 0.2 s, reader 8.1 s, second read 0.2 s, context 0.2 s, report 2.1 s, precedents unavailable for CXR (no index); 5 findings, 5 claims, one finding per id. Faithfulness and stability are not in the pipeline yet, so faithful stays None and findings are verified on supporting symptoms alone · confirmed · run on this machine

  • 2026-10-07 · P3 · MedGemma CXR box format · RESOLVED for the convention, NOT for quality: a grounding prompt returns JSON with box_2d as [y_min, x_min, y_max, x_max] on a roughly 0-1000 grid, and boxes drawn that way land on the thorax; asked for pixel coordinates it still answers on the ~1000 grid, so pixel prompts are wrong. On one chest film (services/medgemma/fixtures/smoke_cxr.png, anatomy only, no lesion) the heart box was too small and too high and the right and left lung boxes were identical, so box IoU against the specialist is not trustworthy yet · confirmed · one-off probe, overlay inspected by eye; the reader keeps not requesting boxes

  • 2026-10-07 · P1 · MedSigLIP (google/medsiglip-448) · gated; accessible with a personal read token after accepting the terms. Loads with transformers.AutoModel in transformers 5.19. Image tower input 448 px, embedding dimension 1152, text length 64, logit_scale parameter exp = 10.03; 3,513,309,984-byte model.safetensors. Embedding takes about 9 s per image on 6 CPU threads · downloaded model and runs.

  • 2026-10-07 · P1 · transformers 5.x get_image_features and get_text_features return an output object, not a tensor; its pooler_output, L2-normalised, equals the model's own image_embeds and text_embeds (checked on a public tiny random SigLIP). The embedder accepts both forms · test_router_real.py.

  • 2026-10-07 · P1 · MedSigLIP preprocessing assumption (448 px, mean and std 0.5, bilinear stretch) was applied by our own prepare, not by the model's processor; embeddings behaved sensibly (zero-shot 95.6%), but equality with the processor output was not checked.

  • 2026-10-07 · P1 · FracAtlas download · the plan's figshare file id 43283628 is outdated; article 22363012 now serves FracAtlas.zip (338,436,460 bytes, CC BY 4.0) as file 65518038 · figshare API.

  • 2026-10-07 · P1 · Brain Tumor Dataset (Cheng) · figshare article 1512427, CC BY 4.0, four ~215 MB zips of MATLAB v7.3 files with cjdata/image and cjdata/PID; used for brain MRI router images · figshare API.

  • 2026-10-07 · P1 · ISIC 2018 S3 archives support HTTP range requests (images read without full downloads); Task 1 validation ground-truth masks are ISIC_xxxx_segmentation.png · direct reads. NLM Montgomery CXR PNGs are readable per file with a browser user agent; the directory index returns 403.

  • 2026-10-07 · P1 · Windows: curl fails TLS revocation checks and the Hugging Face and torchxrayvision downloaders stalled; large files were fetched with curl --ssl-no-revoke -C - (TLS chain still verified). The hf-xet path stalled after 1 GB.

  • 2026-10-07 · P1 · P2's committed model_meta.json formats · classifier preproc_spec is a name (brain_effnet, skin_cls); segmenter preproc_spec is a dict (name, size, channels, mean, std, resize, value_range) and threshold 0.5; detector imgsz 640 and weights loadable with ultralytics.YOLO. ultralytics 8.4.174 builds yolo26n.yaml and its checkpoints load through BoneReader.from_dir · tests/readers/test_real_architectures.py.

  • 2026-10-07 · P1 · Chest model on 12 public Montgomery films (top two findings each, 18 total): median deletion drop at 10% was 0.012 (max 0.116); no finding passed the faithfulness test and no random region did either. Findings were mostly near their operating threshold (0.50 to 0.63), so this says the Grad-CAM++ regions are not demonstrably causal for weak findings, not that the test is broken · verify/results/faithfulness_cxr.json.

  • 2026-10-07 · P1 · Saliency sanity on the chest model, one real film, class Cardiomegaly, seed 0: mean absolute Spearman after cumulative randomisation gradcam 0.0004 (empty in 10 of 11 stages), gradcam++ 0.436 (empty in 5 of 11), hirescam 0.0004; chosen gradcam++ · verify/results/saliency_sanity_cxr.json.

  • 2026-10-07 · P1 · Router, 320 public images (cxr 40 Montgomery, brain 61 Brain Tumor Dataset, skin 100 ISIC, bone 59 FracAtlas, other 60 CIFAR-10), group split 70/15/15: held-out n=50, accuracy 1.00 (Wilson 95% 0.93 to 1.00), every class present; zero-shot over all 320: 0.956 (0.928 to 0.974), bone confused with cxr 5 times and CIFAR with brain 6 times · intake/results/*.json.

  • 2026-10-07 · P1 · OOD on the same set, held-out test: AUROC 1.0 for each of cxr, brain, skin, bone; flagged in-distribution rate cxr 0.17 (1 of 6), brain 0.09, skin 0.20, bone 0.0; every out-of-distribution image flagged · intake/results/ood_report.json.

  • 2026-10-07 · P1 · MedSAM zero-shot with reference-mask boxes, 30 ISIC 2018 Task 1 validation lesions: mean Jaccard 0.878, mean Dice 0.934, failures (Jaccard below 0.65) 0 of 30 · verify/results/medsam_isic_val.json.

Shared log (append-only, newest at the bottom)

2026-10-06 · P1 · WP P1.1, P1.2, P1.3

Did: DICOM/PNG/JPG decode with rescale, windowing and MONOCHROME1 inversion; shared preprocessing specs; eight seeded perturbations; PHI scrub with receipt; quality gate with fix text per failure; intake stage adapter returning StageResult. State: P1.1 to P1.3 done. Perturbation library done early (P1.11 logic still todo). Verified: cd backend && python -m pytest -q gives 121 passed. Includes a pydicom-built MONOCHROME1 fixture, a PHI sentinel test on the serialized bytes, and six degraded images each giving the right reason. Next: CXR reader and anatomy zones (P1.6) so P2.7 can calibrate; then router. Decisions: see Decisions, entries dated 2026-10-06 for P1.

2026-10-06 · P3 · WP P3.1, P3.3, P3.4 (schema), P3.9/P3.10 (design)

Did: Groq pool (cache, RPM/TPM budgeter, retry-after backoff, repair retry, fallback); MedGemma read service (loader, API, batch, Kaggle/Colab notebooks); note schema with injection taxonomy; slot grammar and firewall spec. State: P3.3 done. P3.1 code done, real-model smoke NOT run (no HF_TOKEN here). P3.4 schema done, generator todo. Verified: .venv/Scripts/python -m pytest ml backend services -q -> 48 passed, 1 skipped (the gpu-marked smoke); mypy --strict backend/medproof/llm clean. make smoke does not exist yet (P4.1). Next: run smoke on the local RTX 4060 once HF_TOKEN is set; check CXR box format; gen_notes.py; P3.2. Decisions: see Decisions section (7 entries). Contract change requests: none.

2026-10-07 · P1 · WP P1.6

Did: TorchXRayVision DenseNet121 reader with Grad-CAM family localization (own implementation), PSPNet anatomy masks, patient-side zone naming, contract Finding and ImageEvidence mapping, run(ctx) stage. Switched CXR preprocessing to the library's centre-crop convention after reading its source. Fixed a Grad-CAM++ inversion found by the stub tests. State: P1.6 done. Not yet measured: localization quality (pointing game, box IoU vs RSNA boxes) and faithfulness; those are P2.7 and P1.9. Tiers and the 0.5 threshold are provisional. Verified: cd backend && python -m pytest -q gives 191 passed, 0 skipped, including 5 tests on the real downloaded weights (labels, forward pass, preprocessing parity against the library, end-to-end stage, 14-channel PSPNet). A one-off run on a real public chest film confirmed left/right mask placement and a cardiac-region name for Cardiomegaly; image not stored. Next: P1.4 router, P1.9 faithfulness, P1.11 flip-rate logic. Decisions: see Decisions and Verified facts log, entries dated 2026-10-07. Contract change requests: none.

2026-10-07 · P2 · WP P2.1, P2.2, P2.3 to P2.6 (notebooks)

Did: resumable checksummed downloader with a pinned lock file; pHash leakage audit and group-aware splits; split files, reports/leakage.json and eval_index.json for fracatlas and ham10000; cluster-bootstrap CIs and metrics; shared training code; four Kaggle notebooks (brain classifier, brain segmenter, skin classifier, bone detector) plus a builder and a model registry script. State: P2.1 doing (credential-free sets done; brain, LGG, RSNA wait for the Kaggle token). P2.2 doing (brain, LGG, RSNA splits wait for data). P2.3 to P2.6 blocked: notebooks written and tested, launch needs the Kaggle token and a pushed p2/data branch. Verified: python -m pytest ml/tests -q gives 86 passed (brain and segmenter notebooks run end to end on synthetic data; bone notebook smoke-run for one epoch on CPU). Short real skin run on the laptop GPU (3 epochs, 32 s/epoch, same notebook source, throwaway sanity check, not a Kaggle job): official ISIC 2018 test balanced accuracy 0.697 [0.650, 0.745], melanoma recall 0.667. The full 15-epoch run will be higher. Next: Kaggle token, push p2/data, launch the four jobs, build brain/LGG/RSNA splits, then P2.7 (CXR calibration) and the eval harness. Decisions: see Decisions, entries dated 2026-10-06 for P2. Contract change requests: none. .gitignore carve-out and ml/Makefile.inc need a P4 ack (see Decisions).

2026-10-07 · P4 · WP P4.1, P4.2 (G0)

Did: root Makefile (setup, test, smoke, contracts, stub dev/eval/demo; includes ml/Makefile.inc); root pyproject.toml (ruff + mypy spanning backend/ and ml/); added ruff/mypy to backend/pyproject.toml's dev extra; .pre-commit-config.yaml + .secrets.baseline; minimal web/ (Vite + React + TS) so pnpm install/pnpm dev work and contracts.ts has a home. Contract: reviewed schemas.py against every actual import in backend/ and left it untouched (purely additive freeze); added core/vocab.py and core/status_rule.py (compute_status, precedence rejected > discordant > uncertain > verified); scripts/export_contract.py → contracts/{finding,study_result}.schema.json; scripts/gen_contracts_ts.mjs → web/src/contracts.ts; one hand-written StudyResult fixture per core modality in contracts/fixtures/. State: P4.1 and P4.2 done. Verified: cd backend && python -m pytest -q → 213 passed, 1 skipped, 1 failed (the failure is pre-existing and unrelated: tests/readers/test_anatomy.py needs torchxrayvision, not installed on this machine). backend/tests/core (new) → 28 passed. cd web && npx tsc -b and npx eslint . → clean. node scripts/gen_contracts_ts.mjs regenerates contracts.ts byte-identical on a second run. Next: P4.3 pipeline orchestrator (define StudyContext for real here), then P4.4 API + SSE on these fixtures. Decisions: see Decisions, entries dated 2026-10-07 for P4. Contract change requests: none (additive only).

2026-10-07 · P4 · WP P4.3

Did: backend/medproof/pipeline.py (StageSpec, PIPELINE registry, run_study()); core/context.py (StudyContext dataclass), core/config.py (PipelineConfig, env-backed), core/cache.py (StageCache over diskcache, keyed by (input_sha256, stage, version)). Wired the intake stage via run_bytes() (not the generic run(ctx) wrapper, which drops the decoded image) and a modality-dispatching reader step (only cxr exists; anything else degrades honestly). _collect_findings is the first real caller of status_rule.compute_status outside its own unit tests. Per-stage timeout via a non-context-managed ThreadPoolExecutor (first attempt used with ThreadPoolExecutor(...), whose __exit__ blocks on the abandoned thread and silently defeated the timeout — caught by the timeout test itself, fixed to call shutdown(wait=False) explicitly). Added diskcache to backend/pyproject.toml. Documented .env.example's previously-missing MEDPROOF_* vars. State: P4.3 done, with one caveat recorded plainly rather than glossed over: the WP's literal Done-when ("kill MedGemma mid-run") isn't testable yet since no MedGemma stage exists (P3.1 todo). Verified instead with synthetic slow/raising stages plus the real "no reader for this modality yet" case. Verified: cd backend && python -m pytest tests/test_pipeline.py -v → 8 passed. Full suite python -m pytest -q → 221 passed, 1 skipped, 1 failed (same pre-existing unrelated torchxrayvision gap as the previous entry, nothing newly broken). python -m ruff check backend/medproof/pipeline.py backend/medproof/core/ clean. Next: P4.4, API + SSE, wrapping run_study() and streaming each StageResult as it completes. Decisions: see Decisions, entries dated 2026-10-07 for P4. Contract change requests: none (additive only).

2026-10-07 · P4 · WP P4.4, P4.5

Did: core/ledger.py (Ledger: one global hash-chained append-only JSONL, append/head/verify, hash covers ts/actor/event/study_id too, not just the payload); core/store.py (StudyStore, in-memory, not persisted — stated limitation, see Decisions); api/{app,studies,system}.py (FastAPI + sse-starlette): POST /studies, GET /studies/{id}/events (SSE), GET /studies/{id}, POST /studies/{id}/feedback (the only path to D17's ledger-logged doctor feedback), GET /studies/{id}/fhir, GET /metrics, GET /model-cards, GET /ledger/verify. Added one additive parameter to pipeline.run_study (on_stage_result) so the API can push SSE events and ledger entries per stage without pipeline.py itself depending on the ledger. Makefile's dev target now actually starts uvicorn instead of echoing a stub. State: P4.4 and P4.5 done. Found and fixed one real bug while writing the SSE test: StudyStore.stream()'s original "already-terminal, skip the queue" shortcut silently dropped every buffered stage event for a fast-finishing study (routine with a stub reader finishing before the SSE GET even arrives) — fixed by always draining the queue (it holds every event regardless of when a subscriber connects) and only falling back to the stored terminal state once a queue has been fully drained once already. Verified: cd backend && python -m pytest tests/core/test_ledger.py tests/api -v → 14 passed. Full suite python -m pytest -q → 235 passed, 1 skipped, 1 failed (same pre-existing unrelated torchxrayvision gap, nothing newly broken). ruff check on all new/changed files clean. Manual end-to-end sanity on a real running server (not just TestClient): started uvicorn medproof.api.app:app, uploaded a real phantom PNG via curl, fetched the study, posted feedback, confirmed GET /ledger/verify (entries 2 → 3, ok: true throughout). Next: P4.6 (web app pages) and P4.7 (WebGL viewer), the first real UI, built against web/src/contracts.ts and this API. Decisions: see Decisions, entries dated 2026-10-07 for P4. Contract change requests: none (additive only).

2026-10-07 · P4 · WP P4.6, P4.7

Did: scrollytelling landing (ten chapters on one WebGL2 radiograph, prerendered + hydrated, story engine deferred); /read workstation with live (SSE) and sample modes, WebGL2 viewer (R8/RGBA8/R16F, highp window/level, cividis heatmap + iso-contour, hatched mask, anatomy contours, SDF boxes, zoom about cursor, keyboard map), two-way finding/note linking; validation, models, audit, report and 404 pages; additive API routes (image, float16 pixels, artifacts with traversal guard, per-study ledger); scripts/bake_web_assets.py (real reader outputs on a CC0 chest film); Playwright + axe e2e suite (cd web && pnpm run e2e). State: P4.6 and P4.7 done, with the gaps in the P4 role section stated plainly: live findings withheld until P1.9/P1.11 exist, first SSE stage ~720 ms cold (target 500), viewer 60 fps and Lighthouse not measured on real hardware yet. Verified: cd backend && python -m pytest -q → 266 passed. cd web && npx vitest run → 22 passed; npx tsc -b clean; npx eslint . 0 errors; npx playwright test → 17 passed (no-JS article, zero em/en dashes, axe zero serious/critical on landing, prerendered markup, workstation, validation and models; story boots without console errors; ledger tamper detected in the browser; phone and reduced-motion layouts; keyboard map; span highlight under 100 ms). Next: P4.8 Docker + CI around these suites; model warm-up at startup for the first-stage target; P4.9, P4.10. Decisions: see Decisions, entries dated 2026-10-07 for P4 (landing scope, GSAP licence, two entries, sticky stages, article mode, brain image source, red text shade, browser ledger caveat, content-addressed caches). Contract change requests: none (API routes additive; schemas.py untouched).

2026-10-07 · P3 · WP P3.1, P3.2, P3.4-P3.7, P3.9, P3.10

Did: slot renderer + firewall + model-chosen frames with offline fallback; seeded 200-note generator; injection guard (regex, per-sentence Prompt Guard, classifier) and span-grounded extraction; contradiction rules; report-label mapping, MedGemma client, concordance; real MedGemma reads on the local GPU; eval scripts that recompute from cached output. State: P3.3, P3.4, P3.7, P3.9, P3.10 done. P3.1/P3.2 doing (eval batch reads running; concordance.json is partial until they finish). P3.5/P3.6 measured below. Verified: .venv/Scripts/python -m pytest backend ml/tests/test_notes_schema.py ml/tests/test_gen_notes.py ml/tests/test_eval_second_reader.py ml/tests/test_eval_contradictions.py services -q -> all pass (ml/ tests that need pandas/sklearn/torch are P2's and were not run in my light venv). Numbers (62 held-out test-split notes, live Groq, cached): extraction exact-span F1 0.919 (P 0.922, R 0.917), lenient 0.973, 0 failures; weakest types laterality 0.79, history 0.80, device 0.82. Guard on the corpus 15/15 attacks, 0/47 false positives (regex is in-sample). Guard on 28 fresh attacks / 15 fresh benign: regex 4/28, Prompt Guard@0.9 4/28, classifier 25/28 (1 FP), all three 27/28 (1 FP). Differential: 0 facts overlap the injected text; F1 0.954 on injected notes vs 0.936 on their clean twins. Contradiction rules P/R 1.00/1.00 on gold facts (templated notes: rules fire as designed, not a generalisation claim). Firewall: 104 planted bad claims all blocked, 10 good claims none blocked. Second reader on bone (275-read prefix, reports/concordance.json): sensitivity 0.19 (0.09-0.32), specificity 0.995, kappa 0.27 (0.12-0.43). Next: finish the batch reads and write reports/concordance.json; retrieval (needs P1's embeddings); entailment judge; end-to-end context stage on extracted facts; wire into P4's pipeline. Decisions: see Decisions (7 entries dated 2026-10-07). Contract change requests: none.

2026-10-07 · P3 · WP P3.11, P3.12, P3.13

Did: entailment judge (second wall behind the firewall); Whisper transcription in the Groq pool plus voice dictation through the guard and extraction; FHIR bundle export. Pulled P2's skin-classifier push and wrote the specialist-vs-MedGemma comparison from its saved official-test predictions (agreement, kappa, and whether agreement predicts that the specialist is right); it runs when the HAM10000 reads are in. Found and fixed a crash in the long MedGemma batch (see Decisions). State: P3.11, P3.12, P3.13 done and pushed. HAM10000 reads have only just started, so the skin comparison has no numbers yet; FracAtlas reads about 70% done. Verified: pytest backend + ml/tests (mine) + services: 560+ passed, 5 skipped (the gpu-marked smoke and P2 tests that need torch/sklearn). Entailment: first pass 55/60; fixed judge 59/60 on the fresh variant (29/30 true sentences kept, 30/30 defective caught), 60/60 on the tuned variant; false rejections are the judge's main error (about 2-3%). Voice: real round trip OK. FHIR: bundle validates; 11 tests. Next: run python -m ml.eval_p3.eval_second_reader when the reads finish, then report whether specialist agreement predicts specialist correctness (D13 input for P2); retrieval needs P1's embedding function; wire stages into P4's pipeline. Decisions: see Decisions (4 entries dated 2026-10-07 added). Contract change requests: none.

2026-10-07 · P1 · WP P1.4, P1.5, P1.7, P1.8, P1.9, P1.10, P1.11, P1.12

Did: modality router (MedSigLIP embeddings, temperature-scaled probe, zero-shot fallback, routing before the quality checks); per-modality Mahalanobis OOD score with a cross-fitted threshold, plus an energy score that was tested and rejected; faithfulness (deletion, insertion, permutation test against random regions); stability (eight perturbations, flip rate); saliency sanity (cascading randomisation); MedSAM box-to-mask service with a cache; brain and skin classifier reader, U-Net and MedSAM mask attachment, bone detector wrapper; a script that checks the trained models once their weights are in place. State: P1.4, P1.5, P1.9 to P1.12 done and validated on real models or data. P1.7 and P1.8 blocked: P2's trained weight files are not in the repository (only metadata and predictions); the code is checked on the real timm, segmentation-models and ultralytics architectures with random weights and on P2's committed metadata. Verified: cd backend && python -m pytest -q with the real-model environment set gives 694 passed, 0 skipped. Without the local model folder the real-model tests skip with a reason. Real-data results: router held-out 50 of 50 (CI 0.93 to 1.00), zero-shot 0.956 on 320; OOD AUROC 1.0 held-out; faithfulness 0 of 18 on 12 chest films (random 0 of 18); stability 0 flips at severity 2; MedSAM mean Jaccard 0.878 on 30 lesions. Next: place P2's weights, run python scripts/check_trained_models.py, then close P1.7 and P1.8; retrain router and OOD on the full datasets; repeat faithfulness, stability and saliency per modality. Decisions: see Decisions and Verified facts log, entries dated 2026-10-07 (energy score rejected; faithfulness permutation rule; router confidence saturated on easy classes).

2026-10-07 · P2 · WP P2.1 to P2.14 complete

Did: downloaded and audited all datasets (Kaggle brain, LGG and RSNA included); launched and registered the four Kaggle jobs (skin, brain classifier, segmenter, bone; two accounts because of the 2-GPU-session limit); calibration library and per-model calibration; chest-reader validation on RSNA with external and contaminated weights and Grad-CAM localisation scoring; external brain test on BDNeuro; corruption benchmark; trust-signal and subgroup audits; model cards, datasheets, figures, validation report; make eval. State: P2.1 to P2.14 done. P2.10 is partial by design: discordant needs P3's complete second-reader batch and unfaithful needs P1.9; the harness is ready. Verified: cd backend && python -m pytest -q gives 555 passed; python -m pytest ml/tests -q gives 129 passed; python ml/eval/run_all.py --check reproduces reports/metrics.json exactly in about 100 s. Headline: skin official-test balanced accuracy 0.680 [0.634, 0.726]; brain leakage-free accuracy 0.940 [0.921, 0.958], external BDNeuro 0.755 [0.735, 0.775]; segmenter mean per-patient Dice 0.831 [0.716, 0.890]; bone image AUROC 0.923 [0.885, 0.957]; chest AUROC on RSNA 0.785 (chex) and 0.749 (mimic_ch) against 0.875 for the contaminated all-data weights. Next: P1 to refit the quality gate and add the brain_seg spec; P3 to finish the second-reader batch; P4 to render reports/metrics.json and the cards. Decisions: see Decisions, entries dated 2026-10-07 for P2. Contract change requests: none.

2026-10-07 · P1 · sync and integration check

Did: synced with main (fast-forward to 73b4105), untracked the accidentally committed .env, re-checked model artifacts, ran the import check over every module and the full backend suite. State: ml/artifacts still has metadata and predictions only, no weight files, so P1.7 and P1.8 stay blocked. All medproof modules import; P2's evaluation code uses my decode, preprocess, perturbation, CAM and quality interfaces unchanged. core/schemas.py is still the verbatim copy of the section 4 sketch (no contracts/, web/ or pipeline from P4 yet). Verified: cd backend && python -m pytest -q gives 772 passed, 1 skipped after merging P4's API and web work (693 passed before that merge; the skip is the router accuracy test that needs the staged images, all other real-model tests ran). python -m pytest ml/tests -q gives 128 passed, 1 failed: test_eval_reproducible (ml/eval/run_all.py --check reports that reports/metrics.json differs from a fresh recomputation; confirmed with an LF working copy, so it is not a line-ending effect). Running P2's eval here also rewrites eight calibration*.json files with differences around 1e-7 (different numeric library versions on this machine), which I discarded instead of committing; it looks like P2's calibration fit is not bit-for-bit reproducible across machines. This is a P2 reproducibility issue, not a P1 interface problem; P2 may want to round the stored values or pin the library versions. Next: unchanged (see the P1 section). Decisions: none new. Contract change requests: none.

2026-10-07 · P3 · WP P3.1, P3.2, P3.5, P3.6, P3.8 + red-team

Did: merged the latest main into p3/medgemma-groq; committed the pending work (cue scan, one-call reasoning stage, end-to-end eval, FracAtlas retrieval report); validated the discordant flag against specialist errors for bone and skin; added two guard rules after offline red-teaming; wrote an 83-case offline red-team suite; built and scored the HAM10000 precedent index; refreshed concordance.json from the cached reads; created a CUDA venv and resumed the HAM10000 batch. State: P3.5 and P3.6 done. P3.1 and P3.2 doing (HAM10000 batch running). P3.8 doing (FracAtlas and HAM10000 indices; brain and CXR skipped, MedSigLIP 403). Verified: cd backend && python -m pytest -q gave 732 passed, 10 skipped; python -m tests.redteam.run_redteam gives 83/83 (notes 19, contradictions 13, injection 25, firewall 17, degradation 9); python -m pytest ml/tests/test_eval_second_reader.py 16 passed. Numbers: bone MedGemma sensitivity 0.18 (0.12-0.26) over 569 reads; bone specialist-vs-MedGemma kappa 0.07; discordant-flag error rate 0.63 vs 0.02, with 83% of correct fracture reports flagged; skin over 531 reads, MedGemma names a lesion on 43%, flag effect 0.11 (-0.01, 0.20) not significant; retrieval precision@5 FracAtlas 0.83 (chance 0.70), HAM10000 0.62 (chance 0.47). make smoke still does not exist (P4.1). Not run: Kaggle and Colab notebooks, CXR box format check, brain and CXR retrieval. Next: refresh concordance when the HAM10000 reads finish; wire into P4's pipeline when StudyContext exists. Decisions: see Decisions (4 entries dated 2026-10-07 added). Contract change requests: none.

2026-10-07 · P3 · WP P3.1, P3.2 (batches complete)

Did: finished the HAM10000 MedGemma batch (1,000 of 1,000 reads, 0 failed) on the local RTX 4060 and refreshed reports/concordance.json. State: P3.1 batch reads complete for both datasets; P3.2 numbers final. Verified: python -m ml.eval_p3.eval_second_reader. HAM10000 (official test batch): MedGemma names a lesion type on 42% of images, balanced accuracy 0.107 (0.081-0.136), melanoma sensitivity 0.115 (0.056-0.177); on the 420 images where it did, specialist accuracy 0.82 vs MedGemma 0.51; agreement 0.53, kappa 0.065 (-0.012, 0.145); specialist accuracy when the readers agree 0.85, disagree 0.79; discordant-flag difference in error rate 0.057 (-0.017, 0.125), not significant, so it must not downgrade skin findings. This replaces the 531-read figures in my earlier entry and role notes. Next: CXR box format check; wire into P4's pipeline when StudyContext exists. Decisions: none new. Contract change requests: none.

2026-10-07 · P3 · WP P3.8

Did: MedSigLIP access came through; fixed the embedder (image processor only, transformers 5 output object) and rebuilt the FracAtlas and HAM10000 precedent indices with it; reports are now named per embedder and compared with BiomedCLIP. State: P3.8 doing. FracAtlas and HAM10000 done with MedSigLIP; brain and CXR waiting on P2's images. Verified: cd backend && python -m pytest tests/retrieval -q 13 passed; builds ran on the local RTX 4060. FracAtlas precision@5 0.844 vs chance 0.704 (fracture class 0.476); HAM10000 0.667 vs chance 0.469, balanced vote accuracy 0.486 vs 0.362 for BiomedCLIP, melanoma precision 0.430. Next: brain and CXR retrieval once P2's images are available; wire into P4's pipeline when StudyContext exists. Decisions: none new. Contract change requests: none.

2026-10-07 · P1 · WP P1.7, P1.8

Did: checked ml/artifacts after syncing (origin/main had no new commits): weights.pt now present locally for bone_det, brain_cls, brain_seg and skin_cls, sha256 equal to model_meta.json for all four. Ran check_trained_models.py and a new scripts/faithfulness_trained.py on staged public images. State: P1.7 and P1.8 done. Brain and skin classifiers, the U-Net and the bone detector load through the pipeline's reader code unchanged and give real findings, boxes and masks. Verified: cd backend && python -m pytest -q 1016 passed, 2 skipped. Faithfulness (deletion permutation test) and stability on staged images: over every staged image, brain_cls 69 findings on 59 images, 42% faithful, 20% unstable, mean top-10% drop 0.40; skin_cls 107 findings on 100 images, 44% faithful, 13% unstable, mean drop 0.18 (backend/medproof/verify/results/faithfulness_trained.json). Many brain and skin findings fail the faithfulness test, so faithful=false and the firewall's tighter handling will apply to them; this is a measured property of the trained models on the staged public images (not P2's held-out test sets). Bone gives box evidence only, no heatmap, so faithfulness does not apply to it. Next: P4 wiring; refit thresholds when P2 shares validation images. Skin MedSAM masks were later checked with the trained skin weights on one staged image (masks produced, no flags). Decisions: none new. Contract change requests: none.

2026-10-07 · P3 · WP P3.2, P3.6, P3.8, adapters

Did: pipeline stage adapters (second_read, context, report, precedents) with merge_findings and collect_claims for the orchestrator; stopped second-reader disagreement from demoting findings in unvalidated modalities (P4's status rule reads agrees and box_iou); canonical labels so vocabulary names match dataset codes; lexicon derived from core/vocab.py; a third held-out injection set and its measurement; real MedGemma reads for the four sample images in demo/medgemma_reads; 14 more red-team cases (96 in total). State: P3.2 done. P3.6 measured again, zero-false-positive target not met. Adapters complete on the P3 side and waiting on Contract change request P3-1. Verified: cd backend && python -m pytest -q gives 993 passed, 12 skipped; python -m tests.redteam.run_redteam 96/96; mypy --strict clean on reasoning.py and reasoning_stages.py. Guard third set: regex 6/26, Prompt Guard 0.9 9/26, classifier 25/26, all layers 26/26 with 2/16 benign flagged. Not run: the stages inside a live P4 pipeline (needs P3-1); brain and CXR retrieval; Kaggle and Colab notebooks. Next: P4's reply on P3-1; then a live end-to-end study. Decisions: see Decisions (6 entries dated 2026-10-07 added). Contract change requests: P3-1.

2026-10-07 · P3 · WP P3.8

Did: P2 supplied the download route for brain and RSNA images; downloaded the brain set, verified it against P2's split by coverage and perceptual hash, built and scored the brain precedent index with MedSigLIP, and added brain to the precedent stage. State: P3.8 doing (FracAtlas, HAM10000, brain done; CXR waits for RSNA). Verified: cd backend && python -m pytest tests/test_reasoning_stages.py tests/redteam -q 131 passed; brain precision@5 0.918 vs 0.273 chance, balanced vote accuracy 0.952. The pinned archive size did not match (228,676,924 vs 164,670,110) but 60 of 60 sampled image hashes equal P2's. Not run: RSNA download (needs the competition rules accepted on Kaggle for the token's account), CXR retrieval, MedGemma box quality. Answer to P2: my discordant results have been on main since commit 5a986ca (reports/concordance.json, datasets..specialist_vs_medgemma.discordant_signal), so the metrics and validation report can be regenerated now; I did not need the signals.compare wrapper after 6801d15 but kept it. Next: RSNA download and CXR retrieval once the rules are accepted; P4's reply on P3-1. Decisions: none new. Contract change requests: none new.

2026-10-07 · P4 · CCR P3-1

Did: applied P3-1. StudyContext.stage_results; per-id finding merge (latest stage wins) and claim collection in the orchestrator; P3's stage_specs() registered after the reader behind an ImportError guard; GET /studies/{id}/fhir builds P3's bundle for a finished study (409 while running); workstation shows unvalidated second reads as inconclusive and counts firewall claims; report page lists unblocked claims and downloads the FHIR bundle; start.bat installs .[services]. State: P3-1 done and acked. Live run on the CC0 chest film (real reader, P3's demo second read): six stages, five merged findings with the second read on each, FHIR 200. Not exercised here: note evidence and claims (no Groq key on this machine), so P3 should confirm a live study with keys. Verified: new orchestrator tests (stage_results visible, merge without duplicates, claims collected, parity with merge_findings, registration order and no caching) and FHIR route tests; full backend suite cd backend && python -m pytest -q 998 passed, 8 skipped; cd web && npx vitest run 25 passed, npx tsc -b and npx eslint . clean, npx playwright test 17 passed. Next: P3's end-to-end check with keys; reader warm-up at startup (16.7 s cold against a 20 s timeout); P4.8. Decisions: see Decisions, P3-1 entry dated 2026-10-07. Contract change requests: P3-1 acked and applied.

2026-10-07 · P2 · optional extras

Did: MILK10k external skin test and skin-tone audit; two extra skin seeds on Kaggle and a 3-model ensemble with flip averaging; RAPS, Mondrian and randomised-APS comparison; fixed the Mondrian set rule; added the ML datasheet for MILK10k; make eval now includes the external skin results and the ensemble (about 2 minutes). State: all optional extras from plan.md section 7.2 are done except MedSigLIP linear probes (needs a Hugging Face token). Verified: python ml/eval/run_all.py --check reproduces reports/metrics.json exactly; cd backend && python -m pytest -q and python -m pytest ml/tests -q pass. Next: P1 decides whether the product serves the ensemble; P1 refits the quality gate. Decisions: see Decisions, entries dated 2026-10-07 for P2 (including the exposed Kaggle token). Contract change requests: none.

2026-10-07 · P3 · WP P3.7, P3.9, P3.10

Did: ran a real end-to-end study on the chest sample through P4's pipeline (with the P3-1 wiring on main) and found that a bare side word counted as supporting evidence for every finding; fixed it in the contradiction rules, the firewall and the template report; added a red-team case. P1's trained-model faithfulness results (brain 3 of 12 findings faithful, skin 7 of 13) mean the firewall will withhold most image-only claims for those modalities once faithfulness is wired in. State: P3.7 and P3.10 corrected. Chest weights are present on this machine, so P1's real-weights tests now run. Verified: cd backend && python -m pytest -q gives 1020 passed, 6 skipped; red-team suite 97/97 (policy 3/3); mypy --strict clean on medproof/report; ruff clean on the files I changed. Not run: end-to-end on bone, skin and brain (their trained weights are not on this machine); CXR retrieval and box quality (RSNA images). Next: P4 wiring of faithfulness and stability so faithful gets set; RSNA download once the Kaggle rules are accepted. Decisions: see Decisions (2 entries dated 2026-10-07 added). Contract change requests: none new.

2026-10-07 · P3 · handoff

Did: assigned the RSNA-dependent work (CXR precedent index and the MedGemma chest-box test) to P2, who already has the images; recorded the commands and outputs under the P2 handoff in my role section. State: P3.8 waits on P2 for CXR; the box-quality question stays open until P2 reports. Verified: not run by me (RSNA download on this machine reached about 700 MB of 3.9 GB at 0.3 MB/s). Both scripts have tests: python -m pytest ml/tests/test_build_retrieval.py ml/tests/test_eval_medgemma_boxes.py 18 passed. Next: P2's reports; then PRECEDENT_DATASET gets cxr and I decide on box IoU. Decisions: none. Contract change requests: none.

2026-10-07 · P3 · trained weights, end-to-end on bone, skin and brain

Did: verified and placed P2's trained weights; ran my four stages with the real bone, skin and brain readers on the demo images through P4's run_study (the readers loaded by hand, because the pipeline does not register them); raised the precedents stage timeout to 90 s after a cold-start timeout on brain. State: stages work with all four modalities' real findings. The pipeline itself still needs the three readers registered (P4 and P1). Verified: bone 17.8 s with a located finding, precedents and a claim; skin 20.2 s, finding rejected and no claim; brain 35 s, finding verified on the note's headaches with a claim and a precedents timeout that the new limit covers; python -m pytest tests/test_reasoning_stages.py tests/redteam 133 passed. P1's check_trained_models.py could not run here because it needs sample images staged under backend/data/router, which only exist on P1's machine. Not run: the faithfulness and stability stages on these findings (not in the pipeline), RSNA retrieval and box quality (assigned to P2). Next: P4 and P1 register the readers; then a live study per modality through the API. Decisions: none new. Contract change requests: none.

2026-10-07 · P3 · WP P3.2, P3.8 (RSNA)

Did: downloaded RSNA, built the chest precedent index (MedSigLIP, calibration split as the index), measured MedGemma's chest boxes in two prompt variants, added chest to the precedent stage, and withdrew the handoff to P2 because the work is done. State: P3.8 done for all four modalities. The box question is closed: MedGemma's boxes are not usable. Verified: python -m pytest ml/tests/test_build_retrieval.py ml/tests/test_eval_medgemma_boxes.py passes; cd backend && python -m pytest tests/test_reasoning_stages.py tests/redteam 133 passed; a real chest study through P4's pipeline now returns 5 precedents per finding (first call 28 s while MedSigLIP loads). Chest retrieval precision@5 0.666 against 0.357 for chance; MedGemma boxes 0 of 100 opacities when it may decline, and no better than a fixed central box when forced. Not run: faithfulness and stability on these findings (not in the pipeline), live studies for brain, skin and bone through the API (readers not registered). Next: P4 and P1 register the readers and wire faithfulness and stability; then a live study per modality. Decisions: none new. Contract change requests: none.

2026-10-07 · P4 · Wiring P1's stages

Did: wired P1's router, the brain/skin/bone readers, faithfulness (top 3 findings by default) and stability into PIPELINE per P1's recipe; config gained models_root and faithfulness_top_k; reader no longer cached, verify stages cached with the modality in the key; verify stages read merged findings so neither overwrites the other; workstation copy no longer says faithfulness is pending. State: done for chest end to end (live run: three findings verified, two withheld as untested, 51 s). Brain, skin and bone are wired and tested with stubs only; this machine has no P2 weights. Per P1's latest verify/results/faithfulness_trained.json (59 brain and 100 skin images), 42% of brain findings (29 of 69) and 44% of skin findings (47 of 107) pass faithfulness (the earlier 12- and 13-finding sample gave about a quarter and a half), so more than half of brain and skin findings will be withheld unless notes support them. Bone findings carry boxes but no heatmap, so P1's deletion test returns "not assessable" for them and they stay withheld unless notes support them; P1 to confirm whether that's intended. Verified: tests/test_pipeline_wiring.py 11 passed (router gating and flags, missing-weights degradation per modality, bundle paths, faithfulness and stability on the same finding, cache reuse, modality in the key); full backend suite 1016 passed, 8 skipped; cd web && npx vitest run 25 passed, npx playwright test 17 passed. Next: run a brain/skin/bone study on a machine with P2's weights; startup warm-up; serving-time calibration with P2's Calibrator. Decisions: see Decisions, P1-wiring entry dated 2026-10-07. Contract change requests: none (schemas.py untouched).

2026-10-07 · P3 · live check after P4's pipeline wiring

Did: merged P4's wiring of the router, the brain, skin and bone readers, faithfulness and stability; pushed all P3 work to main (6a90173); ran every demo image through the default pipeline. State: all four modalities complete end to end with my four stages. One design issue found for P4 (an unfaithful finding verified by a non-specific symptom). Verified: cd backend && python -m pytest -q 1032 passed, 6 skipped after the merge; live runs as in the Decisions entry. The shown claims all cite a real note quote, none cites a side word, and flagged note text never appears. Not run: studies through the HTTP API and the web app; calibration is still not in the pipeline (findings carry the flag uncalibrated). Next: P4's reply on the status-rule recommendation; P4 and P2 wiring calibration. Decisions: see the live-run entry. Contract change requests: none.

2026-10-07 · P3 · demo preparation

Did: pulled everything, built the website, ran the API, the MedGemma service and the site, and drove the live flows in a browser (sample, DICOM upload, report, audit, validation, models; 0 browser errors). Found and fixed three things the real inputs exposed: a fitting symptom verified findings whose heatmap had failed the faithfulness test; claims used dataset codes and an unspecified-region phrase; a laterality conflict was reported at full confidence. Wrote the demo guide and a 16-file input pack with a warm-up script. State: all four modalities run end to end through the default pipeline with my four stages; the website works; demo guide in demo/DEMO_GUIDE.md. Verified: cd backend && python -m pytest -q 1051 passed, 6 skipped before the last small change (report package 354 passed after it); red-team 97/97; python demo_inputs/warm_pack.py all 16 files ok; Playwright flows with Edge: landing, workstation, live sample 18.6 s, DICOM upload 22.4 s, report with FHIR download, audit Verify chain. Not working or not shown: automatic routing (Not sure); the website does not display precedents; findings still carry the flag uncalibrated; the quality gate warns but does not lower confidence. Next: P1 to check the router set-up; P4 and P2 to wire calibration; push these commits. Decisions: see the five entries above. Contract change requests: none new.

2026-10-07 · P4 · UI audit and serving-time calibration

Did: audited every backend output against the UI. Added to the workstation: quarantined note instructions, missing-history prompts, finding flags in words, quality reasons with fixes, selected-finding span ownership, a "Run live analysis" button on samples (earlier today), failed regions drawn in red. Added P2's figures to Validation and the datasheets to Models. Wired P2's calibrators as a calibration stage (brain, skin, bone; chest uncalibrated with reason) and made tier abstain uncertain in the status rule. State: done. Live on this laptop with P2's weights and Groq keys: chest 3 verified + 2 uncertain with 5 claims; brain glioma verified on the note (region failed, drawn red); bone fracture verified on the note after calibration; skin nevus withheld. Not shown because not produced here: precedents, router/OOD, masks, voice dictation. Verified: tests/test_calibration_stage.py 7 passed; full backend suite 1023 passed, 8 skipped (one order assertion updated for the new stage); cd web && npx vitest run 33 passed, npx playwright test 17 passed; browser checks of the injection upload, the four Run-live cases and the Validation figures. Next: team decision on note-only verified (see P4 section); brain segmenter / MedSAM masks; P4.8. Decisions: see Decisions, entries dated 2026-10-07 (abstain tier, chest uncalibrated). Contract change requests: none.

2026-10-07 · P4 · Remaining work assigned

Did: assigned the items nothing on the demo laptop produces yet: P1-A masks (U-Net and MedSAM hooks in the reader run() functions), P1-B box-region faithfulness for bone, P1-C FYI on the brain heatmap; P3-A precedents (MedSigLIP and indices; HAM10000 and RSNA thumbnails not displayable under the image-licence rule), P3-B router and OOD artifacts, P3-C voice dictation contract, P3-D the note-only "verified" decision, P3-E report wording. Details and "done when" in the P4 section. State: P4 wiring for each is ready to follow as soon as it lands. Verified: interfaces read from readers/segmenter.py, segment/medsam.py, context/voice.py, retrieval/index.py, intake/router.py. Next: wire each delivery; P4.8. Decisions: none new. Contract change requests: none.

2026-10-07 · P4 · Correction to the assignments

Did: MedSigLIP lives on P1's machine, so router and OOD set-up moved from P3 (old P3-B) to P1 (P1-D), and P1 also hands over the MedSigLIP folder that P3's precedents stage needs (P1-E). P3-A now covers only the retrieval indices and the thumbnail licences. State: assignments updated in the P4 section; nothing else changed. Verified: n/a (documentation). Next: wire each delivery. Decisions: none new. Contract change requests: none.

  • 2026-10-07 · P1 · Heatmap evidence is marked unfaithful when it is off the segmented tumour (heatmap_off_target, under 20% of salient pixels inside the U-Net mask) · the deletion test passed on a heatmap at the image border while the tumour was elsewhere (demo brain image); across 16 brain slices no CAM variant puts more than 14% of its mass in the mask. Threshold provisional; the U-Net is trained on a different dataset.
  • 2026-10-07 · P1 · Box-only findings (bone) get a box-region deletion test against equal-area random boxes, boxes over 35% of the image not assessable · same permutation rule and thresholds as the heatmap test, so one definition of "faithful".

2026-10-07 · P1 · WP P1-A to P1-E (P4's assigned work)

Did: mask wiring for the brain, skin and bone reader stages; box-region faithfulness for bone; off-target heatmap rule for brain; router, OOD and model-folder hand-over (scripts/fetch_models.py). State: P1-A, P1-B, P1-D, P1-E done; P1-C investigated (cause is the classifier, not the CAM method) and handled with a flag. Verified: cd backend && python -m pytest -q 1057 passed, 2 skipped; real trained models: brain heatmap + U-Net mask, skin heatmap + MedSAM mask, bone bbox + MedSAM mask; bone box faithfulness 6 of 7 on staged fracture films; router and OOD run from the exported MedSigLIP folder. Next: P4 passes unet= and medsam= from the pipeline and loads both once at start-up; P2 may want to look at why the brain classifier ignores the tumour (possible dataset shortcut). Decisions: two new, in Decisions. Contract change requests: none.

2026-10-07 · P3 · answers to P4's requests P3-A, P3-C, P3-D, P3-E

Did: packaged the four precedent indices and licence-gated thumbnails (retrieval_pack.zip, docs/retrieval_indices.md); added dictation_to_note (a transcript appends to the typed note) and verified the audio key live; decided P3-D (failed region withheld, untestable region capped at uncertain, filed as P3-2); fixed the report wording (region kept, cough no longer supports a mass). State: all four requests answered; P3-2 waits for P4's one-line change. Verified: cd backend && python -m pytest -q gives 1119 passed, 4 skipped; the P3 ml tests (retrieval build, box evaluation, second reader) 37 passed; a live dictation round trip through Whisper on Groq returned the merged note; thumbnails built (2,447 bone, 3,349 brain); mypy --strict clean on medproof/report and reasoning_stages. Not run: the browser after these changes (the API and website were stopped by the system for low memory, so the thumbnails and the new sentences have not been seen on screen), and P4's side of P3-2. Next: P4 ack on P3-2, the /transcribe route and the notes-box flow, and showing bone and brain thumbnails. Decisions: see the three entries above. Contract change requests: P3-2.

2026-10-07 · P1 · WP P1-A, P1-D (P4's side of the wiring, done by P1 at the team's request)

Did: pipeline.py now passes the mask models to the reader stages (brain gets the U-Net, skin and bone get MedSAM; loaded once per process, a failed load is remembered for five minutes, MEDPROOF_MASKS=0 switches masks off); StudyResult.ood_score now carries the router's score instead of the 0.0 placeholder; fixed _router_step, which flagged every routed study ood because the router's ood is a dict that is always truthy: only is_ood: true flags now. State: P1-A and P1-D done end to end. Through the live API with the router on and no modality given: brain, skin and bone uploads were routed correctly, each got a Mask layer (served as image/png from /studies/{id}/artifacts/), and the brain finding carries heatmap_off_target. The bone finding gets faithful=true from the new box test. Verified: cd backend && python -m pytest -q 1069 passed, 2 skipped. Env for a laptop that runs it: MEDPROOF_ROUTER=1, MEDPROOF_ROUTER_MODEL=<medsiglip folder>, MEDPROOF_MEDSAM_MODEL=<medsam folder>, plus router_probe.npz and ood.npz in backend/artifacts/ (see the P1 section for hashes). MedSAM on CPU is slow; the live runs took 4 to 9 s per study here because the first calls were warm. Next: P4 to warm the mask models at start-up (first MedSAM call loads about 375 MB); P4 to check the UI shows the Mask layer and the heatmap_off_target flag. Decisions: none new. Contract change requests: none. Files outside P1's area touched at the team's request: backend/medproof/pipeline.py (stage wiring, router flag fix, ood score) and a new backend/tests/test_pipeline_masks.py.

2026-10-07 · P3 · voice check

Did: checked the whisper and voice path at every layer; fixed an extraction gap found by it (spelled-out history statements were dropped). State: the voice library works; the API and the website have no voice input yet (P4's to add). Verified: voice unit tests 34 passed; live Whisper run on two synthesized dictations (clean and with a spoken attack): transcripts correct, attack flagged, facts correct including history after the fix; cd backend && python -m pytest -q 1139 passed, 4 skipped; held-out extraction F1 0.922 exact, 0.985 lenient. Not run: a microphone recording in a browser (there is no recording code), and real human speech (the test audio is synthesized). Next: P4 adds the transcribe route and the record button. Decisions: see the two entries above. Contract change requests: none.

2026-10-07 · P4 · Deliveries checked and wired

Did: checked every assigned item on main; applied P3-2 in core/status_rule.py (+4 tests); added POST /transcribe (P3's dictation_to_note, 25 MB limit, audio not stored) and a Dictate button in the notes box; added the "similar confirmed cases" panel (thumbnails only where P3's licence gate supplied one); fixed stage caching so verify results are keyed on their input findings; labelled an off-target heatmap as such instead of blaming the lesion box; start.bat points at local MedSAM/MedSigLIP folders; installed segmentation-models-pytorch and sentencepiece, fetched MedSAM. State: live on this laptop: brain glioma uncertain (heatmap off the U-Net lesion, note supports it, mask drawn), skin nevus withheld (mask drawn), bone fracture verified by its box (mask drawn), chest 3 verified with region-naming sentences. Not testable here until files are copied: router/OOD and precedents (MedSigLIP refused for this token's account; probe files and retrieval pack are with P1 and P3). P1 edited pipeline.py directly for P1-A/P1-D; reviewed and kept. Verified: full backend suite 1129 passed, 7 skipped, 1 failed (test_real_architectures.py::test_bone_reader_loads_a_real_yolo_checkpoint...: needs a newer ultralytics than this laptop's 8.3.221 to build yolo26n.yaml; the trained bone reader itself loads and runs); cd web && npx vitest run 34 passed, npx playwright test 17 passed; browser checks of brain and bone with masks. Next: copy MedSigLIP, the two router files and the retrieval pack to this laptop, then check routing and precedents live; team to pick the demo machine. Decisions: see Decisions (cache key digest, P3-2 applied). Contract change requests: P3-2 acked and applied.

2026-10-07 · P3 · P4's open items

Did: found that P4 had already delivered the transcribe route, the Dictate button, the similar-cases panel and the P3-2 status cap, so verified them with real audio instead of rebuilding them; then closed P4's known gap 0 (cold start) with a startup warm-up that also wakes the MedGemma service. State: voice works end to end in the API and the browser; cold start fixed; nothing of P4's listed items is left that I can do without their files. Verified: cd backend && python -m pytest -q 1164 passed, 4 skipped; API tests 44 passed; /transcribe with real speech; Dictate button in Edge with a fake microphone; a never-seen chest film 109 s to 50 s after a cold restart. Not run: real human speech; the warm-up on a machine with less memory. Next: P4 may want warm-up status in the UI; the team still has to pick one demo laptop. Decisions: see the two entries above. Contract change requests: none.

2026-10-07 · P4 · WP P4.10 (README for the submission)

Did: wrote README.md against the submission guidelines: what Parallax does, the data pipeline (stage table, status rule, API routes, diagram), core models with ids, evidence and where the doctor sees it, a real sample input and output (demo/sample_output/: input, StudyResult, stage trace with ledger hashes and timestamps, FHIR bundle from a live chest run), validation headline with 95% CIs from reports/metrics.json, scope note (MVP vs stretch vs partial), technologies, install, model files, configuration, run (start.bat and manual), reproduction commands, demo order, layout, licences, limitations, team roles. Six screenshots in docs/screenshots/ (CC0, CC BY and public-domain images only). State: P4.10 done. Every number is from reports/metrics.json, the committed sample run, or the code. Verified: all paths the README references exist; sample output has no local filesystem paths. Next: team to decide how the trained weights are distributed for reproduction (e.g. a GitHub release), then link it from the README's "Model files" table. Decisions: none new. Contract change requests: none.