Question about evaluation

[Tran-Nam]

How can run manual evaluation?

My local folder structure:
base
├── evaluation.py
├── init.py
├── metrics.py
├── test_data.py
├── tests
│ └── test-1.zip
When i run: python evaluation.py inside base/, the script return score of model. But when i submit test-1.zip, the status is Error Occured: 'NoneType' object has no attribute 'forecast'. I think this error occurs due to system can not find location of predict.py, which define in prepare.py

What is the script that system uses to evaluate? Can submit system show more information to easier debug?

[ramber1836]

Please follow the instructions described in the Section "Submitted Files" in the Manual (https://github.com/PaddlePaddle/PaddleSpatial/blob/main/apps/wpf_baseline_gru/kddcup22-sdwpf-evaluation/README.md)

[Tran-Nam]

As I understand, my submit.zip will be put inside folder tests/, and when run evaluation, the file location tests/submit.zip will be pass to evaluate function?

[ramber1836]

Please read the released evaluation.py carefully, it demonstrates all the details of the evaluation.
BTW, the submitted zip file will be placed in a temporary location.

[luckyyangrun]

I have met the same error... How do you solve it？

[ramber1836]

As mentioned in the readme "Manual on Evaluation"

The files to be submitted are illustrated as follows:

./your-folder-name (rename it acording to your setting)
| --- init.py
| --- predict.py (required) (rename it acording to your setting)
| --- prepare.py (required) (DO NOT rename it)
| --- ... (optional, other related scripts)
| --- ./your-model-folder (optional) (rename it acording your setting)
| --- ...

Say, for instance, if the files to be submitted were located in a folder which is named 'your-folder-name', then, the files in this folder (i.e. 'your-folder-name') should include:

1. a prediction-like python script, e.g. predict.py
2. a prepare.py script
3. other stuff, like, your code that the prediction script depends on, the cached model you have already trained on your local machine, etc.

You do not need to submit the test data and the evaluation scripts, since these are already placed for the online tests.

[Tran-Nam]

We must load in TestData inside file "pred_file"? How can we import test_data.py from "pred_file" if we dont know location of our folder?

[ramber1836]

Please refer to the evaluation.py script for more details. The logic is that the test_data (including test_x and test_y) have already been loaded if the pred_file was invoked by the evaluation script.

…

On Sun, May 15, 2022 at 9:43 AM phuongnam ***@***.***> wrote: We must load in TestData inside file "pred_file"? How can we import test_data.py from "pred_file" if we dont know location of our folder? — Reply to this email directly, view it on GitHub <#54 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEVHY35WVE3TFAQEVLQ4VZ3VKBJD7ANCNFSM5V2EKOZQ> . You are receiving this because you commented.Message ID: ***@***.*** com>

[ForestsKing]

I have the same problem, how did you solve it? Thank you!

[Tran-Nam]

I found that my code import torch but I submit in framework base.