Skip to content

Use type-6 quantiles for simulation-based CIs (fixes #288)#289

Closed
fabian-s wants to merge 1 commit into
masterfrom
issue-288
Closed

Use type-6 quantiles for simulation-based CIs (fixes #288)#289
fabian-s wants to merge 1 commit into
masterfrom
issue-288

Conversation

@fabian-s

@fabian-s fabian-s commented Jun 10, 2026

Copy link
Copy Markdown
Collaborator

Closes #288.

Problem

All simulation-based CIs computed interval bounds as empirical alpha/2 / 1 - alpha/2 quantiles of nsim posterior coefficient draws via stats::quantile() with its default type = 7. For B draws, E[F(X_(k))] = k/(B+1), so type-7 bounds at B = 100 enclose an expected central mass of only ~93.1% instead of the nominal 95% — the intervals are systematically too narrow even if the posterior approximation were perfect (see #288 for the full derivation and the Monte Carlo evidence from the #285 study).

Changes

  • quantile(..., type = 6) (plotting positions k/(B+1), which remove the systematic inward bias) at all six sim-CI sites: get_sim_ci(), get_sim_ci_cumu(), get_sim_ci_surv() (i.e. add_hazard() / add_cumu_hazard() / add_surv_prob() with ci_type = "sim"), get_cif.default() (add_cif()), add_trans_ci() (add_trans_prob() — a site not listed in the issue), and compute_cumu_diff() (get_cumu_coef()).
  • Documentation: the ci_type/nsim docs now state the type-6 construction, that the default nsim = 100 gives noisy bounds (recommend 500+), and that nsim < 2/alpha - 1 (39 for alpha = 0.05) cannot achieve the nominal level at all. NEWS entry notes that all sim-CI bounds widen slightly relative to <= 0.7.5.
  • Tests: new seed-reproduction regression test verifying the bounds equal type-6 quantiles of the posterior draws for the hazard, cumulative-hazard, and survival paths (at nsim = 100, exercising the interpolation branch); four hardcoded snapshot values updated (all widen, as the math predicts).

Default nsim values are deliberately unchanged (100, add_cif() already at 500): the type fix removes a bias, raising nsim reduces Monte Carlo noise — an orthogonal, runtime-relevant decision better made separately (issue #288, point 2).

Validation

Beyond the unit tests, the fix was validated empirically by rerunning part of the #285 Monte Carlo study (attic/simulations/ci-surv-prob on branch claude/confident-cori-dng5at; nominal-95% CIs for S(t|x), not included in this PR — verification artifacts and protocol in results-type6/, commit 7e6a41c): scenarios 1, 30, 33 — the three with the worst sim100 coverage in the full 500-rep run — x 100 reps with the patched package. Since the study's RNG streams are deterministic per (scenario, rep), the rerun is exactly paired with reps 1-100 of the stored type-7 results: point estimates and the (unaffected) delta/default bounds agree to <= 1e-4, so only the quantile type differs.

method type-7 coverage type-6 coverage gain theory predicts cells flipping miss→cover / cover→miss median width ratio
sim, nsim = 100 0.907 0.928 +2.1pp ~+1.9pp 59 / 2 1.081
sim, nsim = 500 0.919 0.925 +0.6pp ~+0.4pp 16 / 1 1.016

Gains match the order-statistics prediction within Monte Carlo error, and coverage flips are essentially one-directional (type-6 intervals contain the type-7 intervals for the same draws). Residual under-coverage in these worst-case cells (heavy censoring, late times) is the late-hazard fit bias documented in the #285 study — it affects delta and sim CIs alike and is out of scope here.

R CMD check: 0 errors, 0 warnings, 4 pre-existing notes (untouched files). Full test suite passes.

🤖 Generated with Claude Code

stats::quantile()'s default type 7 made all simulation-based CI bounds
systematically too narrow at small nsim: at the default nsim = 100 the
expected central mass of the draws between the bounds is ~93% instead
of the nominal 95%. Type-6 quantiles (plotting positions k/(B+1))
remove this systematic inward bias.

Applied at all six sim-CI sites: get_sim_ci(), get_sim_ci_cumu(),
get_sim_ci_surv(), get_cif.default(), add_trans_ci(), and
compute_cumu_diff(). Documents the achieved level and nsim sensitivity
in the ci_type/nsim docs, adds a seed-reproduction regression test for
the hazard/cumulative-hazard/survival paths, and updates four hardcoded
snapshot test values (intervals widen slightly at both ends).

Closes #288

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings June 10, 2026 14:22

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates simulation-based confidence interval construction across pammtools to use stats::quantile(..., type = 6) for posterior-draw quantiles, addressing systematic under-coverage at small nsim (fixes #288).

Changes:

  • Switch all simulation-based CI bounds to type-6 empirical quantiles in hazard / cumulative-hazard / survival, CIF, transition probabilities, and cumulative-coefficient differences.
  • Expand user-facing documentation to describe the type-6 construction and the impact of small nsim on Monte Carlo noise/achievable nominal level.
  • Add a regression test that reproduces posterior draws via seeding and verifies bounds match type-6 quantiles; update existing snapshot expectations accordingly.

Reviewed changes

Copilot reviewed 4 out of 10 changed files in this pull request and generated no comments.

Show a summary per file
File Description
tests/testthat/test-add-functions.R Adds a seed-reproduction regression test for type-6 sim-CI bounds; updates expected sim-CI snapshot values.
R/cumulative-coefficient.R Uses type = 6 quantiles for simulated CIs in compute_cumu_diff().
R/add-functions.R Switches all sim-CI quantile calls to type = 6 (hazard/cumu/surv/CIF/transitions) and documents the behavior.
NEWS.md Notes the behavioral change (slightly wider sim-CIs) and links it to #288.
man/get_hazard.Rd Documents type-6 sim-CI quantiles and nsim guidance/limits.
man/get_cumu_hazard.Rd Documents type-6 sim-CI quantiles and nsim guidance/limits.
man/compute_cumu_diff.Rd Updates CI description to specify type-6 empirical quantiles.
man/add_trans_prob.Rd Documents type-6 quantiles for simulated transition-probability intervals.
man/add_hazard.Rd Documents type-6 sim-CI quantiles and nsim guidance/limits.
man/add_cif.Rd Documents type-6 quantiles for CIF simulated intervals.
Files not reviewed (6)
  • man/add_cif.Rd: Language not supported
  • man/add_hazard.Rd: Language not supported
  • man/add_trans_prob.Rd: Language not supported
  • man/compute_cumu_diff.Rd: Language not supported
  • man/get_cumu_hazard.Rd: Language not supported
  • man/get_hazard.Rd: Language not supported

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

fabian-s added a commit that referenced this pull request Jun 10, 2026
Reruns scenarios 1, 30, 33 (worst sim100 coverage in the full run) x 100
reps with the patched package from PR #289, exactly paired with reps
1-100 of the stored type-7 results via the deterministic per-(scenario,
rep) RNG streams. Pairing verified (delta/default bounds agree to
<= 1e-4); sim coverage moves 0.907 -> 0.928 (nsim = 100, theory ~+1.9pp)
and 0.919 -> 0.925 (nsim = 500), with near-one-directional coverage
flips. See results-type6/README.md for the protocol.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@johannespiller johannespiller left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review ok. Ready to merge.

@fabian-s

Copy link
Copy Markdown
Collaborator Author

see #292

@fabian-s fabian-s closed this Jun 15, 2026
fabian-s added a commit that referenced this pull request Jun 17, 2026
Consolidated PRs #284, #287, #289, #290, #291 and consistent t=0 boundary rows, shared simulation draws, and S3 dispatch fixes for `add_*` functions
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Sim-based CIs use type-7 sample quantiles: systematically anti-conservative at the default nsim = 100

3 participants