Discrepancy of the model performance in KITTI dataset

[DKandrew]

Hi Shashank,

I run the infer-main.ipynb. By default, it is using INFER-Skip (Top 5) in the KITTI dataset, but the result that I get is

1s: 2.209328362278883, 2s: 2.9157824738774303, 3s: 3.683025655408532, 4s: 5.294841714296214

I believe this is the same ADE metric that you used in Table 1 of your paper, but why the ADE here is significantly worser than what your paper recorded? How should I replay the result in the paper?

Regards,
DK

[DKandrew]

It seems like after training the network, using the checkpoint_future.tar from ablation_cache/skipLSTM/30_04_2020_10_46/split-0 we can get a better result than the pre-trained model, but it is still worse than what the paper reported.

1s: 2.327015556371463, 2s: 2.8690266160160203, 3s: 3.1390985630001627, 4s: 3.795339987032598

[talsperre]

Hi, these errors reported above are in the pixel space. To get the error in (m), we need to multiply the same by 0.25 since our grid resolution is 0.25m. Also, the results for KITTI are computed using cross-validation so the results are averaged over the splits.

[fgqile]

Hi !
When I retrained on the data you provided, the results I found were not as good as those provided in your paper. The following are the settings of each parameter. Do you have any suggestions?
my computer system is ubuntu16.04

expID split-0
dataDir /data/1lpx/INFER-master/data/INFER-datasets/kitti
csvDir /data/1lpx/INFER-master/data/INFER-datasets/kitti/final-validation
trainPath /data/1lpx/INFER-master/data/INFER-datasets/kitti/final-validation/train0.csv
valPath /data/1lpx/INFER-master/data/INFER-datasets/kitti/final-validation/test0.csv
optMethod adam
initType default
lossFun default
modelType skipLSTM
activation relu
groundTruth True
augmentation True
scaleFactor False
minMaxNorm False
batchnorm False
softmax False
lane True
obstacles True
road True
vehicles True
lr 0.0001
dilation 1
augmentationProb 0.3
momentum 0.9
beta1 0.9
beta2 0.9
gamma 1
weightDecay 0
gradClip 10
seqLen 1
imageWidth 256
imageHeight 256
futureFrames 20
futureEpochs 10
nepochs 60
channels ['lane', 'obstacles', 'road', 'vehicles']

the result is as follows