Batch Processing
batch_find_untargeted_attack
enables users to run find_adversarial_example
for multiple samples from a single dataset, writing 1) a single summary .csv
file for the dataset, with a row of summary results per sample, and 2) a file per sample containing the output dictionary from find_adversarial_example
.
batch_find_untargeted_attack
allows verification of a dataset to be resumed if the process is interrupted by intelligently determining whether to rerun find_adversarial_example
on a sample based on the solve_rerun_option
specified.
Index
MIPVerify.batch_find_targeted_attack
MIPVerify.batch_find_untargeted_attack
MIPVerify.run_on_sample_for_targeted_attack
MIPVerify.run_on_sample_for_untargeted_attack
Public Interface
MIPVerify.batch_find_untargeted_attack
— Methodbatch_find_untargeted_attack(
nn,
dataset,
target_indices,
optimizer,
main_solve_options;
save_path,
solve_rerun_option,
pp,
norm_order,
tightening_algorithm,
tightening_options,
solve_if_predicted_in_targeted,
adversarial_example_objective
)
Runs find_adversarial_example
for the specified neural network nn
and dataset
for samples identified by the target_indices
, with the target labels for each sample set to the complement of the true label.
It creates a named directory in save_path
, with the name summarizing
- the name of the network in
nn
, - the perturbation family
pp
, - the
norm_order
Within this directory, a summary of all the results is stored in summary.csv
, and results from individual runs are stored in the subfolder run_results
.
This functioned is designed so that it can be interrupted and restarted cleanly; it relies on the summary.csv
file to determine what the results of previous runs are (so modifying this file manually can lead to unexpected behavior.)
If the summary file already contains a result for a given target index, the solve_rerun_option
determines whether we rerun find_adversarial_example
for this particular index.
optimizer
specifies the optimizer used to solve the MIP problem once it has been built and main_solve_options
specifies the options that will be passed to the optimizer for the main solve.
Named Arguments:
save_path
: Directory where results will be saved. Defaults to current directory.pp, norm_order, tightening_algorithm, tightening_options, solve_if_predicted_in_targeted
are passed through tofind_adversarial_example
and have the same default values; see documentation for that function for more details.solve_rerun_option::MIPVerify.SolveRerunOption
: Options arenever
,always
,resolve_ambiguous_cases
, andrefine_insecure_cases
. Seerun_on_sample_for_untargeted_attack
for more details.
Internal
MIPVerify.batch_find_targeted_attack
— Methodbatch_find_targeted_attack(
nn,
dataset,
target_indices,
optimizer,
main_solve_options;
save_path,
solve_rerun_option,
target_labels,
pp,
norm_order,
tightening_algorithm,
tightening_options,
solve_if_predicted_in_targeted
)
Runs find_adversarial_example
for the specified neural network nn
and dataset
for samples identified by the target_indices
, with each of the target labels in target_labels
individually targeted.
Otherwise same parameters as batch_find_untargeted_attack
.
MIPVerify.run_on_sample_for_targeted_attack
— Methodrun_on_sample_for_targeted_attack(
sample_number,
target_label,
summary_dt,
solve_rerun_option
)
Determines whether to run a solve on a sample depending on the solve_rerun_option
by looking up information on the most recent completed solve recorded in summary_dt
matching sample_number
.
summary_dt
is expected to be a DataFrame
with columns :SampleNumber
, :TargetIndexes
, :SolveStatus
, and :ObjectiveValue
.
MIPVerify.run_on_sample_for_untargeted_attack
— Methodrun_on_sample_for_untargeted_attack(
sample_number,
summary_dt,
solve_rerun_option
)
Determines whether to run a solve on a sample depending on the solve_rerun_option
by looking up information on the most recent completed solve recorded in summary_dt
matching sample_number
.
summary_dt
is expected to be a DataFrame
with columns :SampleNumber
, :SolveStatus
, and :ObjectiveValue
.
Behavior for different choices of solve_rerun_option
:
never
:true
if and only if there is no previous completed solve.always
:true
always.resolve_ambiguous_cases
:true
if there is no previous completed solve, or if the most recent completed solve a) did not find a counter-example BUT b) the optimization was not demosntrated to be infeasible.refine_insecure_cases
:true
if there is no previous completed solve, or if the most recent complete solve a) did find a counter-example BUT b) we did not reach a provably optimal solution.