Skip to content

test(canary): parametrise canary detection across QMIA and LiRA#462

Merged
jim-smith merged 1 commit into
452-worstcase-report-individualfrom
449-parametrise-canary-test
May 26, 2026
Merged

test(canary): parametrise canary detection across QMIA and LiRA#462
jim-smith merged 1 commit into
452-worstcase-report-individualfrom
449-parametrise-canary-test

Conversation

@shamykyzer

@shamykyzer shamykyzer commented May 25, 2026

Copy link
Copy Markdown
Contributor

closes #449

  • i extracted the canary scaffolding into a shared canary_target fixture in tests/conftest.py that returns (target, canary_idx, n_train)
  • the fixture builds the target with data generation, 9-NN boundary selection, label flip on 8 rows, and a bootstrap=False RandomForest, same as the original QMIA only test
  • new file tests/attacks/test_canary_predictions.py holds a parametrised test_attack_predicts_canaries over QMIA and LiRA
  • each attack row carries its own score field (member_prob vs score), AUC threshold (0.90 vs 0.85, looser for shadow model attacks per the issue body), and canary count threshold (7/8 vs 6/8)
  • the original test_qmia_predicts_canaries is removed to avoid drift, with a short comment in test_qmia_attack.py pointing at the new home
  • WorstCase is intentionally not included yet, its per record output (added in WorstCaseAttack to expose per-record scores #452) indexes into an internal train_test_split of the combined train+test predictions, so the original training set's canary_idx does not map directly into instance_0["individual"]["member_prob"]
  • a dedicated WorstCase canary test that walks the split indices is a follow up, scoped out here to keep this change focused
  • this branch is stacked on 452-worstcase-report-individual so future WorstCase inclusion has the per record output available
  • note the diff against main will be larger than the actual test: parametrise canary test across attacks (qmia, lira, worstcase) #449 change until both feat: MetaAttack model #441 and WorstCaseAttack to expose per-record scores #452 merge, after which a rebase will shrink it

@shamykyzer shamykyzer requested a review from jim-smith May 26, 2026 14:39
@jim-smith jim-smith merged commit 217474f into 452-worstcase-report-individual May 26, 2026
1 check passed
@jim-smith jim-smith deleted the 449-parametrise-canary-test branch May 26, 2026 15:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants