Test harness extensibility

[Ka-zam]

#813 points to a fundamental flaw in the testing of kernels.

What about including something like this in the kernel header file and then generating input accordingly?

  #if 0 // VOLK_TEST_SPEC
  domain: [0x00800000, 0x7F7FFFFF]
  sweep_negative: false
  tolerance: 8e-6
  comparison: absolute
  edge_cases: [-1.0, 0.0, inf, 65536.0]
  benchmark_sizes: [1024, 65536, 131071]
  #endif

It's unconventional but would be effective.

[marcusmueller]

I'd counter-propose we'd rather a separate struct that contains functions that generate the respective test points or offer evaluation functions. And since the test fixtures are C++, this simply can be a C++ class. (and instead of #if 0 simply go #if VOLK_TEST_FIXTURE to make the compiler implement that, or move it to a separate .cc file)

By the way, I still like that! If you're really investing in going that way: unused C strings aren't "expensive". You could just put JSON in a string with a kernel-specific name.

[marcusmueller]

Btw, the idea of having a C struct in the .h that describes the kernel – in-placeness, input/output types, available architectures, actual granularity of implementations, is older, I think. It would allow us to throw away all the stupid number of points % SIMD width != 0 handling in each kernel, and let consumers – especially GNU Radio, which often can choose whether it actually needs to ask for the sine of 1025 items, or whether it can ask for 1024 now and then care about the 1 leftover when there's more input later.

I don't think mixing test fixture with description of the kernel is wise, but there's definitely overlap in what a test fixture needs to know about the kernel and what a consumer of the kernel would want to know, and the fact that you're starting with domain: hints at that being a long-missing puzzle piece.

[Ka-zam]

I'll work on this and come up with a suggestion. I think a C struct is the way to go!

[jdemel]

We have quite a lot of kernels that do a simple mathematical operation on individual elements. This is the source structure for our tests. However, we have quite a few special kernels that are flaky because we don't test them correctly. That's why I introduced gtest. I'd like to extend that set. Maybe there's a way to combine those ideas =D