feat: MetaAttack model

CHANGELOG.md

@@ -3,12 +3,30 @@
 ## [Unreleased]
 
 Changes:
+*   Feat: `MetaAttack`: aggregate per-record vulnerability across multiple privacy attacks (LiRA,
+    QMIA, Structural) into a unified vulnerability DataFrame with within-attack (mean, std,
+    consistency) and cross-attack (arithmetic/geometric MIA mean, structural flag, total
+    vulnerability count) aggregation. Supports three operating modes via `behaviour`:
+    `'run_all'` (fresh execution), `'use_existing_only'` (collate from pre-existing
+    `report.json` files without re-running — critical for attacks such as LiRA that may
+    take weeks on large model grids), and `'fill_missing'` (run only attacks not already
+    present). Outputs `vulnerability_matrix.csv` alongside the standard JSON report.
+    By default appends the MetaAttack section to an existing `report_dir/report.json`
+    (set `keep_separate=True` for a standalone file). PDF report includes a bar chart
+    of records grouped by the number of attacks flagging them. `use_existing_only`
+    and `fill_missing` scan both the canonical single-file `report_dir/report.json`
+    (multi-section, as produced when individual attacks append to the same file)
+    and any subdirectory-per-attack layout. Registered in the attack factory as
+    `"meta"`.
 *   Feat: `QMIAAttack`: membership inference attack via quantile regression (Bertran et al.,
     NeurIPS 2023, arXiv:2307.03694). Trains a histogram-based quantile regressor
     (`HistGradientBoostingRegressor`) on non-member hinge scores to learn per-sample
     membership thresholds. A sample is predicted as a member when its observed score
     exceeds the predicted threshold at quantile level (1 - alpha). No shadow models or
     architecture knowledge required. Registered in the attack factory as `"qmia"`.
+*   Fix: `StructuralAttack` now respects the `report_individual` flag. Per-record
+    `record_level_results` and `attack_metrics["individual"]` are only populated when the
+    flag is set to `True`, matching the behaviour of `LIRAAttack` and `QMIAAttack`.
 
 ## Version 1.4.3 (Jan 29, 2026)
 

README.md

@@ -100,6 +100,29 @@ Run the full benchmark comparing QMIA against WorstCase and LiRA:
 python examples/sklearn/benchmark_qmia_full.py
 ```
 
+## MetaAttack: Unified Per-Record Vulnerability Aggregation
+
+`MetaAttack` runs multiple privacy attacks (LiRA, QMIA, Structural) on the same target and aggregates their per-record results into a single vulnerability DataFrame.  Three operating modes are supported via the `behaviour` parameter:
+
+* **`'run_all'`** (default) — run every specified attack from scratch.
+* **`'use_existing_only'`** — read per-record scores from pre-existing `report.json` files without re-running anything.  Useful when expensive attacks such as LiRA have already been run.
+* **`'fill_missing'`** — load existing results and run only the attacks not yet present.
+
+```python
+from sacroml.attacks.meta_attack import MetaAttack
+from sacroml.attacks.target import Target
+
+target = Target(model=model, X_train=X_train, y_train=y_train, X_test=X_test, y_test=y_test)
+meta = MetaAttack(
+    attacks=[("lira", {}), ("qmia", {}), ("structural", {})],
+    behaviour="run_all",  # alternatives: "use_existing_only", "fill_missing"
+    output_dir="output_meta",
+)
+meta.attack(target)
+```
+
+The vulnerability matrix is saved as `vulnerability_matrix.csv` in `output_dir`.
+
 ## Documentation
 
 See [API documentation](https://ai-sdc.github.io/SACRO-ML/).

examples/sklearn/meta_attack_example.py

@@ -0,0 +1,104 @@
+"""Example: run a MetaAttack combining QMIA and structural attacks.
+
+Trains a RandomForest on synthetic data, wraps it in a Target, then
+runs MetaAttack to produce a cross-attack vulnerability DataFrame.
+
+Usage::
+
+    python examples/sklearn/meta_attack_example.py
+"""
+
+import logging
+
+from sklearn.datasets import make_classification
+from sklearn.ensemble import RandomForestClassifier
+from sklearn.model_selection import train_test_split
+
+from sacroml.attacks.meta_attack import MetaAttack
+from sacroml.attacks.target import Target
+
+logging.basicConfig(level=logging.INFO)
+
+output_dir = "output_meta_attack"
+
+if __name__ == "__main__":
+    # --- Prepare target ---
+    X, y = make_classification(
+        n_samples=300,
+        n_features=10,
+        n_informative=5,
+        n_classes=2,
+        random_state=42,
+    )
+    X_train, X_test, y_train, y_test = train_test_split(
+        X, y, test_size=0.4, stratify=y, random_state=42
+    )
+
+    model = RandomForestClassifier(n_estimators=100, random_state=42)
+    model.fit(X_train, y_train)
+
+    target = Target(
+        model=model,
+        dataset_name="synthetic",
+        X_train=X_train,
+        y_train=y_train,
+        X_test=X_test,
+        y_test=y_test,
+        X_train_orig=X_train,
+        y_train_orig=y_train,
+        X_test_orig=X_test,
+        y_test_orig=y_test,
+    )
+    for idx in range(X.shape[1]):
+        target.add_feature(f"feature_{idx}", [idx], "float")
+
+    # --- Run MetaAttack ---
+    meta = MetaAttack(
+        attacks=[
+            ("qmia", {}, 2),  # QMIA with 2 repetitions
+            ("structural", {}),  # Structural (single run)
+        ],
+        behaviour="run_all",  # alternatives: "use_existing_only", "fill_missing"
+        mia_threshold=0.5,
+        output_dir=output_dir,
+    )
+    meta.attack(target)
+
+    # --- Inspect results ---
+    df = meta.vulnerability_df
+
+    print("\n=== Vulnerability Matrix (first 10 records) ===")
+    print(df.head(10).to_string())
+
+    print("\n=== Summary Statistics ===")
+    n_train = int(df["is_member"].sum())
+    n_test = len(df) - n_train
+    print(f"Training records:  {n_train}")
+    print(f"Test records:      {n_test}")
+
+    # MIA vulnerability
+    if "qmia_vuln" in df.columns:
+        n_qmia = int(df["qmia_vuln"].sum())
+        print(f"QMIA vulnerable:   {n_qmia}")
+
+    # Structural vulnerability (training records only)
+    if "struct_vuln" in df.columns:
+        train_df = df[df["is_member"] == 1]
+        n_struct = int(train_df["struct_vuln"].sum())
+        print(f"Struct vulnerable:  {n_struct} (of {n_train} training)")
+
+    # Records vulnerable to all attacks
+    max_attacks = int(df["n_vulnerable"].max())
+    n_all = int((df["n_vulnerable"] == max_attacks).sum())
+    print(f"Vulnerable to all:  {n_all} (flagged by {max_attacks} attacks)")
+
+    # Top-10 most vulnerable training records by MIA mean
+    if "mia_mean" in df.columns:
+        top10 = df[df["is_member"] == 1].nlargest(10, "mia_mean")[
+            ["mia_mean", "mia_gmean", "n_vulnerable"]
+        ]
+        print("\n=== Top 10 Most Vulnerable Training Records ===")
+        print(top10.to_string())
+
+    print(f"\nReport saved to: {output_dir}/")
+    print(f"CSV saved to:    {output_dir}/vulnerability_matrix.csv")

sacroml/attacks/constants.py

@@ -0,0 +1,34 @@
+"""Shared numerical and default-value constants for the attacks package.
+
+Centralising these here avoids duplication across attack modules and makes
+the *why* of each magic number visible at a glance.
+
+Notes
+-----
+A separate :data:`sacroml.attacks.utils.EPS` (``1e-16``) and an identical
+``EPS`` in :mod:`sacroml.attacks.likelihood_attack` are kept independently
+for now because they predate this module and migrating them is a wider
+refactor.  A follow-up PR can converge those onto a single constant defined
+here once the call sites have been audited.
+"""
+
+from __future__ import annotations
+
+EPS_META: float = 1e-10
+"""Tolerance added before ``log()`` in geometric-mean aggregation.
+
+Looser than :data:`sacroml.attacks.utils.EPS` (``1e-16``) because the
+geometric mean of MIA scores in :class:`~sacroml.attacks.meta_attack.MetaAttack`
+does not need the same precision as normal-distribution CDF/PDF
+calculations and benefits from a value comfortably above floating-point
+denormals.
+"""
+
+DEFAULT_MIA_THRESHOLD: float = 0.5
+"""Default cutoff above which a per-record membership-inference score is
+flagged as vulnerable.
+
+Used as the ``mia_threshold`` default for
+:class:`~sacroml.attacks.meta_attack.MetaAttack` so the value can be
+referenced symbolically from tests, examples, and documentation.
+"""

sacroml/attacks/factory.py

@@ -7,6 +7,7 @@
 from sacroml.attacks.attack import Attack
 from sacroml.attacks.attribute_attack import AttributeAttack
 from sacroml.attacks.likelihood_attack import LIRAAttack
+from sacroml.attacks.meta_attack import MetaAttack
 from sacroml.attacks.qmia_attack import QMIAAttack
 from sacroml.attacks.structural_attack import StructuralAttack
 from sacroml.attacks.target import Target
@@ -19,6 +20,7 @@
 registry: dict[str, type[Attack]] = {
     "attribute": AttributeAttack,
     "lira": LIRAAttack,
+    "meta": MetaAttack,
     "qmia": QMIAAttack,
     "structural": StructuralAttack,
     "worstcase": WorstCaseAttack,