An issue regarding to AV2 dataset leaderboard

[xyzmro]

Dear authors, thank you very much for your excellent work!

I faced a problem when I was trying with your codes and weights. I have downloaded the fine-tuned checkpoint 'QCNet-p_1-f.ckpt', and executed the 'test.sh' script to generate the submission file 'submission.parquet'. After I uploaded it to the leader board, which is 'Single Agent Motion Forecasting' competition, it shows that it is failed to execute. May I ask that the 'test.py' and 'test.sh' are currently not supporting competition of AV2 leaderboard upload, please? If they are not supporting AV2 leader board upload, where I can upload the 'submission.parquet' to check the results of test split of AV2 dataset?

Thank you for your help in advance and looking forward for your reply.

[youngzhou1999]

Hi,

Is there any log/print of your submitted results so that we can see why it failed? Also, check the file size of 'submission.parquet'. It's about 75 MB.

[xyzmro]

Hello, apologize for disturbing!

Unfortunately, the AV2 leader board does not show any log/output when I submitted the 'submission.parquet'.
Here is the screenshot:

Besides, based on what you said, the file size of 'submission.parquet' should be 75 MB, but I only generate around 37.5 MB 'submission.parquet', which is really weird.

The 'test.sh' command is:

python test.py --model QCNet --root /path/to/av2 --ckpt_path /path/to/QCNet-p_1-f.ckpt

And the 'QCNet-p_1-f.ckpt' checkpoint file was downloaded from your link.

I will have another try, but anyway, sincerely thank you for your help and reply!

[xyzmro]

I tried again and it is still failed. Here is the AV2 leader board link: https://eval.ai/web/challenges/challenge-page/1719/overview

Any the 'submission.parquet' is still around 37.5 MB.

[youngzhou1999]

Please open these links and see what they show.

[xyzmro]

Sure, after I clicked 'Submitted file', the original 'submission.parquet' will be downloaded.

The 'Result file':

[]

The 'Stdout file':

Submission evaluation failed, see stderr for additional details.

The 'Stderr file':

KeyError('No predictions found for scenario 0000b329-f890-4c2b-93f2-7e2413d4ca5b.')

[youngzhou1999]

It shows that you miss some data for evaluation, which makes sense since your file is only about 40 MB. You may check your dataset first to see whether you have all the data. The val set and test set should have 24988 and 24984 data samples, respectively.

[xyzmro]

May I ask how to get the number of data samples in val set and test set, please?

I tried a Python script to get the number of data samples by simply checking the file with '.parquet' suffix name in folders, but it shows really huge numbers, which are:

The number of data samples in split train is 573567850.
The number of data samples in split val is 71696443.
The number of data samples in split test is 32279808.

Just a quick note is that, after executing the 'val.sh' and 'test.sh', two zip files are downloaded, inside the zip file, there are 'processed' directory and 'raw' directory. However, in the terminal, I haven't seen any log showing the number of data samples.

[youngzhou1999]

You can print the length of self.processed_paths shown in the getitem function https://github.com/youngzhou1999/SmartPretrain/blob/main/datasets/argoverse_v2_dataset.py#L522.

[xyzmro]

Thank you very much for your tips! It works, but the numbers of data samples I got in validation set and test set are 24988 and 24984, respectively. These numbers match the numbers you said before. I am also quite confused about why it does not work now.

To better demonstrate the details of reproduction experiment I take:

I used two A6000 GPUs (48GB VRAM) workstation. The OS is Ubuntu 20.04. I directly clone your repo. After I created a new directory in order to place dataset and downloaded your fine-tuned checkpoints, I executed the 'val.sh' first. I only change the codes related to the path of directory and checkpoint, which are:

python val.py --model QCNet --root /home/mc451742/SmartPretrain/argo2 --ckpt_path /home/mc451742/SmartPretrain/checkpoints/QCNet-p_1-f.ckpt

Then, a 'val.tar' zipped file was downloaded to my 'argo2' directory and extracted automatically. Inside the 'val' folder, they are two sub directories, which are 'processed' and 'raw'. Similarly with 'test.sh' script, I executed it:

python test.py --model QCNet --root /home/mc451742/SmartPretrain/argo2 --ckpt_path /home/mc451742/SmartPretrain/checkpoints/QCNet-p_1-f.ckpt

The zipped file was downloaded and extracted automatically.

The 'val.sh' generate the results in terminal, and the 'test.sh' does not. However, 'test.sh' generate the 'submission.parquet'. I'd like to check the results in official AV2 leaderboard, so I submitted the 'submission.parquet' file, and the issue showed up. Finally I open an issue here to ask. Hope this description could help and sincerely thank you for your helpful assistance. ;)

[youngzhou1999]

I understand why now. It seems that I deleted the key id when loading data. You can comment this line https://github.com/youngzhou1999/SmartPretrain/blob/main/datasets/argoverse_v2_dataset.py#L531.
For testing, this is used to get the ID of the agents. You can see it here https://github.com/youngzhou1999/SmartPretrain/blob/main/predictors/qcnet.py#L323.

[xyzmro]

Well, I tried what you said to comment out this line https://github.com/youngzhou1999/SmartPretrain/blob/main/datasets/argoverse_v2_dataset.py#L531.

The code now is:

But the result in AV2 leaderboard is still failed...

I will give the log reported as well:

[]

Submission evaluation failed, see stderr for additional details.

KeyError('No predictions found for scenario 0008c251-e9b0-4708-b762-b15cb6effc27.')

And, I also tried to modify the code in this line as you said https://github.com/youngzhou1999/SmartPretrain/blob/main/predictors/qcnet.py#L323.

I changed it from:

eval_id = list(compress(list(chain(*data['agent']['id'])), eval_mask))

to

eval_id = list(compress(list(chain(*data['agent'])), eval_mask))

But it is still failed as well, and the log is:

[]

Submission evaluation failed, see stderr for additional details.

KeyError('No predictions found for scenario 0000b329-f890-4c2b-93f2-7e2413d4ca5b.')

I double checked their file size, they are still around 36.5 (37) MB.

The only difference is their execution time, while the former one (only comment out) is around 11 seconds, and the later one (comment out + modification) is around 8 seconds.

Hope my message helps.

[youngzhou1999]

Well, I just noticed that the pop ID logic is inside the pertaining phase and does not affect the test/val phase.

You can try the following:

debug inside the test_end function https://github.com/youngzhou1999/SmartPretrain/blob/main/predictors/qcnet.py#L332 to check the length of self.test_predictions, it should be 24984.

Also, you can debug here https://github.com/youngzhou1999/SmartPretrain/blob/main/predictors/qcnet.py#L322 to see what data['scenario_id'] looks like.