Fluster counts 'not executed' tests as passing

[lumag]

When printing the summary of the tests, the Fluster would count 'not executed' tests as passing, increasing the numer of the passed tests.

[mdimopoulos]

Hi @lumag can you provide an example of a decoder and test suite which seems to be problematic?
I tried to reproduce with
./fluster.py r -d FFmpeg-H.264 -ts JVT-AVC_V1 -s and
./fluster.py r -d FFmpeg-H.264 -ts JVT-Professional_profiles -s
to no avail, e.g.

`|Test|FFmpeg-H.264|
|-|-|
|TOTAL|14/38|
|NOT SUPPORTED|0/38|
|FAIL/ERROR|24/38|
|TOTAL TIME|60.194s|

Profile	FFmpeg-H.264
HIGH_10_INTRA	7/7
HIGH_4_2_2_INTRA	7/7
HIGH_4_4_4_INTRA	0/6
HIGH_4_4_4_PREDICTIVE	0/11
CAVLC_4_4_4	0/3
CAVLC_4_4_4_INTRA	0/4

All of the above using master branch

[lumag]

The following snippet was generated using ec789e0. Maybe I'm misunderstanding the meaning of 'TOTAL' row. My assumtion was that the first number is the number of passed tests. Maybe a correct way would be to add 'SKIPPED' row, which would be hidden if no tests were skipped.

Test	FFmpeg-H.265-v4l2m2m	GStreamer-H.265-V4L2
TOTAL	123/147	126/147
NOT SUPPORTED	0/147	0/147
FAIL/ERROR	24/147	21/147
TOTAL TIME	195.582s	367.161s

Test	FFmpeg-H.265-v4l2m2m	GStreamer-H.265-V4L2
AMP_A_Samsung_7	☠	☠
AMP_B_Samsung_7	☠	☠
AMP_D_Hisilicon_3	✔️	✔️
AMP_E_Hisilicon_3	✔️	✔️
AMP_F_Hisilicon_3	✔️	✔️
AMVP_A_MTK_4	✔️	✔️
AMVP_B_MTK_4	✔️	✔️
AMVP_C_Samsung_7	✔️	✔️
BUMPING_A_ericsson_1	✔️	✔️
CAINIT_A_SHARP_4	✔️	✔️
CAINIT_B_SHARP_4	✔️	✔️
CAINIT_C_SHARP_3	✔️	✔️
CAINIT_D_SHARP_3	✔️	✔️
CAINIT_E_SHARP_3	✔️	✔️
CAINIT_F_SHARP_3	✔️	✔️
CAINIT_G_SHARP_3	✔️	✔️
CAINIT_H_SHARP_3	✔️	✔️
CIP_A_Panasonic_3	✔️	✔️
cip_B_NEC_3	✔️	✔️
CIP_C_Panasonic_2	✔️	✔️
CONFWIN_A_Sony_1	❌	✔️
DBLK_A_MAIN10_VIXS_4	☠	⌛
DBLK_A_SONY_3	✔️	✔️
DBLK_B_SONY_3	✔️	✔️
DBLK_C_SONY_3	✔️	✔️
DBLK_D_VIXS_2	✔️	✔️
DBLK_E_VIXS_2	✔️	✔️
DBLK_F_VIXS_2	✔️	✔️
DBLK_G_VIXS_2	✔️	✔️
DELTAQP_A_BRCM_4	✔️	✔️
DELTAQP_B_SONY_3	✔️	✔️
DELTAQP_C_SONY_3	✔️	✔️
DSLICE_A_HHI_5	✔️	✔️
DSLICE_B_HHI_5	✔️	✔️
DSLICE_C_HHI_5	✔️	✔️
ENTP_A_QUALCOMM_1	✔️	✔️
ENTP_B_Qualcomm_1	✔️	✔️
ENTP_C_Qualcomm_1	✔️	✔️
EXT_A_ericsson_4	✔️	✔️
FILLER_A_Sony_1	✔️	✔️
HRD_A_Fujitsu_3	✔️	✔️
INITQP_A_Sony_1	✔️	✔️
INITQP_B_Main10_Sony_1	☠	⌛
ipcm_A_NEC_3	✔️	✔️
ipcm_B_NEC_3	✔️	✔️
ipcm_C_NEC_3	✔️	✔️
ipcm_D_NEC_3	✔️	✔️
ipcm_E_NEC_2	✔️	✔️
IPRED_A_docomo_2	✔️	✔️
IPRED_B_Nokia_3	✔️	✔️
IPRED_C_Mitsubishi_3	✔️	✔️
LS_A_Orange_2	✔️	✔️
LS_B_Orange_4	✔️	✔️
LTRPSPS_A_Qualcomm_1	✔️	✔️
MAXBINS_A_TI_5	✔️	✔️
MAXBINS_B_TI_5	✔️	✔️
MAXBINS_C_TI_5	✔️	✔️
MERGE_A_TI_3	✔️	✔️
MERGE_B_TI_3	✔️	✔️
MERGE_C_TI_3	✔️	✔️
MERGE_D_TI_3	✔️	✔️
MERGE_E_TI_3	✔️	✔️
MERGE_F_MTK_4	✔️	✔️
MERGE_G_HHI_4	✔️	✔️
MVCLIP_A_qualcomm_3	✔️	✔️
MVDL1ZERO_A_docomo_4	✔️	✔️
MVEDGE_A_qualcomm_3	✔️	✔️
NoOutPrior_A_Qualcomm_1	✔️	✔️
NoOutPrior_B_Qualcomm_1	✔️	✔️
NUT_A_ericsson_5	✔️	✔️
OPFLAG_A_Qualcomm_1	✔️	✔️
OPFLAG_B_Qualcomm_1	❌	✔️
OPFLAG_C_Qualcomm_1	❌	✔️
PICSIZE_A_Bossen_1	☠	☠
PICSIZE_B_Bossen_1	☠	☠
PICSIZE_C_Bossen_1	☠	☠
PICSIZE_D_Bossen_1	☠	☠
PMERGE_A_TI_3	✔️	✔️
PMERGE_B_TI_3	✔️	✔️
PMERGE_C_TI_3	✔️	✔️
PMERGE_D_TI_3	✔️	✔️
PMERGE_E_TI_3	✔️	✔️
POC_A_Bossen_3	✔️	✔️
PPS_A_qualcomm_7	✔️	✔️
PS_B_VIDYO_3	✔️	✔️
RAP_A_docomo_6	✔️	✔️
RAP_B_Bossen_2	⌛	✔️
RPLM_A_qualcomm_4	✔️	✔️
RPLM_B_qualcomm_4	✔️	✔️
RPS_A_docomo_5	✔️	✔️
RPS_B_qualcomm_5	✔️	✔️
RPS_C_ericsson_5	✔️	✔️
RPS_D_ericsson_6	✔️	✔️
RPS_E_qualcomm_5	✔️	✔️
RPS_F_docomo_2	✔️	✔️
RQT_A_HHI_4	✔️	✔️
RQT_B_HHI_4	✔️	✔️
RQT_C_HHI_4	✔️	✔️
RQT_D_HHI_4	✔️	✔️
RQT_E_HHI_4	✔️	✔️
RQT_F_HHI_4	✔️	✔️
RQT_G_HHI_4	✔️	✔️
SAO_A_MediaTek_4	✔️	✔️
SAO_B_MediaTek_5	✔️	✔️
SAO_C_Samsung_5	✔️	✔️
SAO_D_Samsung_5	✔️	✔️
SAO_E_Canon_4	✔️	✔️
SAO_F_Canon_3	✔️	✔️
SAO_G_Canon_3	✔️	✔️
SAO_H_Parabola_1	✔️	✔️
SAODBLK_A_MainConcept_4	✔️	⌛
SAODBLK_B_MainConcept_4	✔️	⌛
SDH_A_Orange_4	✔️	✔️
SLICES_A_Rovi_3	✔️	✔️
SLIST_A_Sony_5	✔️	✔️
SLIST_B_Sony_9	✔️	✔️
SLIST_C_Sony_4	✔️	✔️
SLIST_D_Sony_9	✔️	✔️
SLPPLP_A_VIDYO_2	✔️	✔️
STRUCT_A_Samsung_7	✔️	✔️
STRUCT_B_Samsung_7	✔️	✔️
TILES_A_Cisco_2
TILES_B_Cisco_1
TMVP_A_MS_3	✔️	✔️
TSCL_A_VIDYO_5	✔️	✔️
TSCL_B_VIDYO_4	✔️	✔️
TSKIP_A_MS_3	✔️	✔️
TSUNEQBD_A_MAIN10_Technicolor_2	☠	⌛
TUSIZE_A_Samsung_1	☠	☠
VPSID_A_VIDYO_2	✔️	✔️
VPSSPSPPS_A_MainConcept_1	☠	✔️
WP_A_MAIN10_Toshiba_3	☠	⌛
WP_A_Toshiba_3	✔️	✔️
WP_B_Toshiba_3	✔️	✔️
WP_MAIN10_B_Toshiba_3	☠	⌛
WPP_A_ericsson_MAIN10_2	☠	⌛
WPP_A_ericsson_MAIN_2	✔️	✔️
WPP_B_ericsson_MAIN10_2	☠	⌛
WPP_B_ericsson_MAIN_2	✔️	✔️
WPP_C_ericsson_MAIN10_2	☠	⌛
WPP_C_ericsson_MAIN_2	✔️	✔️
WPP_D_ericsson_MAIN10_2	☠	☠
WPP_D_ericsson_MAIN_2	☠	☠
WPP_E_ericsson_MAIN10_2	☠	⌛
WPP_E_ericsson_MAIN_2	✔️	✔️
WPP_F_ericsson_MAIN10_2	☠	⌛
WPP_F_ericsson_MAIN_2	✔️	✔️

Test	FFmpeg-H.265-v4l2m2m	GStreamer-H.265-V4L2
TOTAL	123/147	126/147
NOT SUPPORTED	0/147	0/147
FAIL/ERROR	24/147	21/147
TOTAL TIME	195.582s	367.161s

[mdimopoulos]

Mmmmm thanks. Let me take a closer look and let you know. TOTAL will have to be changed to PASSED actually, not sure if it was always like this or it was changed recently.

[mdimopoulos]

Are these the vectors that cause the issue you mentioned? Are they being counted as passed?
TILES_A_Cisco_2 TILES_B_Cisco_1

[lumag]

Are these the vectors that cause the issue you mentioned? Are they being counted as passed?
TILES_A_Cisco_2 TILES_B_Cisco_1

Yes, exactly those two.

[mdimopoulos]

Cool. I am looking into it and doing light refactoring. I haven't seen this happen before, i suspect the bug was introduced when we decided to create a test result category for not executed.
I hope to have a fix soon, hopefully tomorrow.

[mdimopoulos]

While investigating this issue i found another one that is not directly related, but it skews the statistics when having a test suite with expected errors (test suite variable negative_test). Basically, skipping vectors is being overridden during execution.
I've fixed both of them and the reporting itself for markdown, i want to do some final tests for csv and json on Monday.

[mdimopoulos]

By the way i noticed that there are a few timeouts in your report, to avoid them i suggest you use less threads with -j <number> and/or increase timeout threshold with -t <num_of_seconds>

[lumag]

By the way i noticed that there are a few timeouts in your report, to avoid them i suggest you use less threads with -j <number> and/or increase timeout threshold with -t <num_of_seconds>

The timeouts are usually caused by the firmware not being able to cope with the stream and failing to signal that to the userspace.

[mdimopoulos]

I believe #360 fixes the bug. Can you take a look at your convenience and let me know?
Thanks in advance

PR has been merged to master, you can take the latest from there if you like