[New Feature Request] Meta-attack

[jim-smith]

Is your feature request related to a problem? Please describe.
Currently each individual attack can calculate and output the vulnerability of each member of the training set into results.json.

However we typically report measures (accuracy, true positive rate st some fixed false positive rate, PDIF/FDIF rates) that aggregate over all the records.

What that doesn't tell us is whether some training records are vulnerable to all (or most of ) the attacks.

• or to repeated runs of the same attack (since some are stochastic)
• Or whether some test records are always misclassified.

Describe the solution you'd like
Something that pulled all the relevant individual vulnerabilities into a pandas data frame - one column per attack, one row per train (and possibly test) record.

• that would give us the ability to produce Venn diagrams.
• or other create a 'meta-attack' that scored 'how many different attacks is this record vulnerable to' or (since the vulnerabilities may be attack confidence values in $[0-1]$ - either the geometric or arithmetic mean of those vulnerabilities.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Start by running one of the simple sklearn examples and when you run lira and structural attacks set report_individual=True in the constructor
that should let you see where the record-level vulnerabilities gt put in results .json

[jim-smith]

@JessUWE this the meta attack we talked about about in the sacro-ml meeting and then later when you and I met to discuss tasks over the next 2-3 weeks

[shamykyzer]

Overview

Architecture flow with behaviour modes

Aggregation pipeline

Class hierarchy (click to expand)