Describe the bug
With iterations or warmup_iterations parameters set > 1, setup, benchmark, and cleanup queries are treated as individual queries. That is, each query will run multiple times before running the next query. For example, given 2 iterations, the execution order will be setup -> setup -> benchmark -> benchmark -> cleanup -> cleanup. This breaks the contract of the setup and cleanup queries. This is relevant for a table format benchmark where the setup operation is modifying tables.
Steps/Code to reproduce bug
Run the nds_power script with iterations or warmup_iterations > 1.
Expected behavior
With iterations or warmup_iterations set > 1, setup, benchmark, and cleanup queries should run as a unit of iteration. For example, given 2 iterations, the execution order should be setup -> benchmark -> cleanup -> setup -> benchmark -> cleanup.
Describe the bug
With iterations or warmup_iterations parameters set > 1, setup, benchmark, and cleanup queries are treated as individual queries. That is, each query will run multiple times before running the next query. For example, given 2 iterations, the execution order will be setup -> setup -> benchmark -> benchmark -> cleanup -> cleanup. This breaks the contract of the setup and cleanup queries. This is relevant for a table format benchmark where the setup operation is modifying tables.
Steps/Code to reproduce bug
Run the nds_power script with iterations or warmup_iterations > 1.
Expected behavior
With iterations or warmup_iterations set > 1, setup, benchmark, and cleanup queries should run as a unit of iteration. For example, given 2 iterations, the execution order should be setup -> benchmark -> cleanup -> setup -> benchmark -> cleanup.