Issues with the SuperTuxKart benchmark

[Alayan-stk-2]

I'm opening a ticket here as that was the only place I found to report issues.

There are several problematic elements with the SuperTuxKart test on OpenBenchmarking:

• By far the biggest issue is the set of Vulkan tests and how Vulkan results are reported. The OpenGL renderer supports several important graphical features that the Vulkan renderer currently does not, with a high impact on both visual quality and performance, such as shadows. There are several graphical parameters that can be set as active, but will only be picked up and used if the active renderer supports them. This is made very clear in-game when switching the renderer (the option to tick on and off all the unsupported options is grayed out) and in the in-game benchmark text (settings not supported by the active renderer are shown as turned off), but this is not picked up in your test. Vulkan results for the "medium", "high" and "ultimate" presets are not comparable to the OpenGL results with the same preset name because of this. It probably doesn't make sense to test the Vulkan renderer on more than one preset.
• Another issue is the presets themselves. I have not examined them in every last detail, but there are multiple apparent problems:
  • One need to go look in the PhoronixTestSuite config entry to find out what each preset mean, as there is no direct correspondence with SuperTuxKart presets. The names themselves can be confusing, as for example the "Low" setting still uses dynamic lighting while the game offers significantly lower settings than that for old or low-end hardware.
  • The Geometry Detail setting, which determines Level-of-Detail distances, is not set.
  • The Ultimate preset uses weaker settings than the highest settings available in STK 1.5. 2048 shadows and PCSS are not enabled, and there may be some other differences. To be fair, I'm not quite sure if the command line option to enable and disable PCSS was added, but the fact that 2048 shadows are not used tells me there was no update to the settings.
  • The choice of what settings are enabled or disabled in the various presets is also questionable and doesn't follow how they are organized in the game's presets. SSAO is a fairly intensive effect that is enabled in the "medium" level on OpenBenchmarking, while the more important shadows effect is only enabled starting with the High effect. In game it is the opposite, shadows are enabled before SSAO.
• Finally, this is not a test configuration issue but more one of results display: the dropdown is extremely long, and part of that is because for each setting, there are two entries, the one for "steady FPS" and the one for "mostly stable FPS". Both sets of results are generated and collected together, as they are simply each a different way to analyze the same benchmark run. If it was possible to display both in the same page, it would reduce clutter in the dropdown and also make it easier to compare both values with one another.