In order to be valid, a benchmark run needs to meet the “95% on-time requirement”. The results_log.csv file contains the actual_start_time and the scheduled_start_time of each of the issued queries. In order to have a valid run, 95% of the queries must meet the following condition:
actual_start_time − scheduled_start_time < 1 second
It would be important to clarify whether the 1 second corresponds to wall clock time or simulation time.
It would be important to clarify whether the 1 second corresponds to wall clock time or simulation time.