Skip to content

Add PCNtoolkit conformance and held-out validation - #4

Merged
bbuchsbaum merged 13 commits into
mainfrom
gauntlet-integration
Aug 25, 2026
Merged

bbuchsbaum merged 13 commits into
mainfrom
gauntlet-integration

Conversation

@bbuchsbaum

@bbuchsbaum bbuchsbaum commented Aug 23, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • Pin PCNtoolkit 1.3.0 fixtures and classify exact, converted, matched-estimator, same-estimand, and non-equivalent comparisons.
  • Make mgcv fit status, model selection, calibration estimands, and held-out assessment fail closed.
  • Bind cross-fit, calibration, longitudinal, frozen-bundle, HBR, and site-transfer behavior to explicit provenance and uncertainty contracts; refuse RDS deserialization unless the caller explicitly acknowledges a trusted source.
  • Add a five-scenario replicated PCNtoolkit matrix, HBR/site-transfer receipts, post-hoc NHANES development evidence, and a separately frozen untouched NHANES confirmation.
  • Correct the public candidate install/status wording, support denominators, site-strategy count, frozen-bundle privacy and hash claims, and clipped diagnostic plots.

Local validation

  • devtools::test(stop_on_failure = TRUE): 1,233 passed, 0 failed, 0 skipped. The two recorded warnings are environment-version warnings.
  • Built referent_0.1.0.tar.gz with all six vignettes, then ran R CMD check --as-cran --no-manual: 0 errors, 0 warnings, 1 expected New submission note.
  • Installed-package Starlight generation rendered all six articles; post-processing, the Astro static build, and the fail-closed checker verified 37 advertised routes by file, H1 identity, canonical URL, redirect state, and image alt text.
  • README quickstart output was reproduced with explicit support counts and an assessment denominator tied to finite held-out scores.
  • Local, tracking, and live branch SHA: b1080373fe22fec3f424d0eb025a3c8405f8a1cd.

Scientific gate outcomes

  • Replicated PCN matrix: linear Gaussian, nonlinear heteroskedastic, unequal-site, and covariate-shift scenarios are equivalent within the registered margins.
  • Skew-heavy: all 20 Referent and comparator fits are valid with no critical comparator warnings. Referent's held-out log-score contrast is 0.396069 (95% replicate-bootstrap interval 0.378490 to 0.412005), calibration guardrails pass, and the registered classification is superior. The seed block and result entered version control together, so this is not described as independently preregistered.
  • HBR/site transport: both HBR stages have zero divergences, maximum R-hat 1.00, and minimum bulk/tail ESS above 1,700. Every site passes separately. Observed-site-only prior selection chose the Referent adaptation priors; independent site 5 has 90% coverage 0.9067, MACE 0.0267, mean Z 0.0563, and variance Z 1.051.
  • NHANES 2017-2018 is explicitly labeled post-hoc model-development evidence because its earlier failure informed the demographic scale terms before the same cohort was re-evaluated. The separately frozen 2013-2014 cohort is the untouched confirmation; all marginal, tail, shape, and conditional gates pass. Neither lane is a survey-weighted national-prevalence estimate.
  • HBR and site receipts bind every retained CSV by SHA-256, row count, and schema. Validation independently recomputes row identities, divergence counts, R-hat/ESS summaries, prediction summaries, prior selection, and all per-site gates rather than trusting receipt booleans.

Release decision

The scoped registered evidence now passes, and the source is prepared as a 0.1.0 release candidate. This does not establish blanket equivalence or dominance over PCNtoolkit: estimator-divergent lanes remain labeled non-equivalent, the skew-heavy claim is limited to its named release matrix, and the NHANES confirmation is limited to the declared unweighted cross-cycle transport contract. Until this PR is merged and a release is tagged, installation and documentation should remain PR-qualified.

cursor[bot]
cursor Bot previously approved these changes Aug 23, 2026

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale comment

Approved. Cursor Security Agent completed with no findings that need human review, and Cursor Bugbot was not running on this PR. No reviewers were assigned.

Open in Web View Automation 

Sent by Cursor Approval Agent: Pull Request Router and Approver

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f1395bf34e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread tools/pcntoolkit/run_hbr_site_evidence.py
Comment thread .github/workflows/pcntoolkit-validation.yml Outdated
Comment thread tools/pcntoolkit/render_evidence.R Outdated
@cursor
cursor Bot dismissed their stale review August 24, 2026 14:36

Dismissed the prior approval because Cursor Security Agent did not finish within the wait window, so this head no longer qualifies. Cursor Bugbot was not running on this PR. Human review is needed; no reviewers were assigned because the only recent editor is the pull request author.

cursor[bot]
cursor Bot previously approved these changes Aug 24, 2026

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale comment

Approved. Cursor Security Agent completed with no findings that need human review, and Cursor Bugbot was not running on this PR. No reviewers were assigned.

Open in Web View Automation 

Sent by Cursor Approval Agent: Pull Request Router and Approver

@cursor
cursor Bot dismissed their stale review August 24, 2026 15:09

Dismissed the prior approval because Cursor Security Agent: Security Reviewer was still pending after 8 minutes. Cursor Bugbot was not running on this PR. No reviewers were assigned.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale comment

Security review of this PR found one high-confidence issue in the new ref_read() interchange path. GitHub Actions and the rest of the statistical/validation surface did not introduce additional reportable issues.

Open in Web View Automation 

Sent by Cursor Security Agent: Security Reviewer

Comment thread R/reference.R
cursor[bot]
cursor Bot previously approved these changes Aug 24, 2026

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale comment

Approved. Cursor Security Agent completed with no findings that need human review, and Cursor Bugbot was not running on this PR. No reviewers were assigned.

Open in Web View Automation 

Sent by Cursor Approval Agent: Pull Request Router and Approver

@cursor
cursor Bot dismissed their stale review August 24, 2026 20:08

Dismissed the prior approval because Cursor Security Agent: Security Reviewer was still pending after the wait window on the current head. Cursor Bugbot was not running. No reviewers were assigned.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. Cursor Security Agent completed with no findings that need human review, and Cursor Bugbot was not running on this PR. No reviewers were assigned.

Open in Web View Automation 

Sent by Cursor Approval Agent: Pull Request Router and Approver

@bbuchsbaum
bbuchsbaum merged commit f20c49f into main Aug 25, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant