Post Checking Predictions

[fresleven]

The conclusions I do are from my own POV and I may not have all the info.

Confusion Matrix

Here, I examine the confusion matrices with two thresholds: 0.5 and 0.62, 0.575, 0.875 on the top and bottom respectively. Each row adds up to 100%, so we want to know for each GT class, how much was predicted as bkg, building, vegetation, or water. The best per-class thresholds using sweep_thresholds.py is 0.650, 0.550, 0.900. I didn't run it since I forgot to have time to do it.

These models are the original gated models with the max height. I believe they still show the same trends on the metrics that we're concerned about: Building IoU, Water IoU, and Building Height.

Plots

Conclusion

• Building IoU: This is likely caused by the prediction threshold, as decreasing the threshold helps per-pixel classification. However, this doesn't correspond to increases of IoU. As such, decreasing the threshold and using some post-processing techniques could help to increase (like having a minimum blob of > 10 pixels. We should be using post-processing techniques on all image segmentation to increase our chances.
• Water IoU: This is also caused by the prediction threshold, but water is more mispredictd as vegetation, as there is more vegetation. Finding a way to counteract the data imbalance should help us with segementation.

GT Height for different folds

We want to examine our training/testing split and figure out why our fold 0 performs similarly to the leaderboard.

Plots

Conclusion

In fold 0, there are a lot more higher buildings. You likely did building_height = torch.exp(building_logits), how did that perform? Also, we may want to consider predicting extremely high buildings and calculating the height separately.

Height Predictions on Fold 0

We analyze how does our predicted height vary with the true height.

Plots

Conclusion

Similar to the previous section, we significantly underpredict tall buildings and vegetation. Our activation function of softplus may not be sufficient to handle tall structures. We should consider new activation functions or ways to predicting taller structures. Vegetation over 30m and buildings over 20m are our failure point.

Worst Predictions on Fold 0

We visualize our worst predictions based on the score and analyze where we went wrong.

Plots

Conclusion

Overall, do to our threshold, we often underpredict. IoU may not correspond to per-pixel classification, and we could have a lower threshold and include some post-processing to straighten it out.

Overall Conclusions

• Change how we predict height (activation function, separating tall vs small buildings, etc..) to predict tall structures.
• Edit how we do sweep_thresholds.py or how we predict probabilities of building/vegetation/water in an area.
• Include post-processing techniques to smooth out our predictions

Questions:

• In the leaderboard metric and/or sweep_thresholds.py, do we calculate total IoU or do we take the mean across different patches?
• What other activation functions or combinations of activation functions can we use for height?
• How do people typically deal with height outliers?
• What postprocessing techniques do you think would help us?

Code

You can run this code for your own models by putting plot_analysis.py in tools/ and doing sbatch plot_analysis.txt

plot_analysis.py

plot_analysis.txt