Taylor Diagrams

[pantoniad]

Discussed in #16

^{Originally posted by pantoniad October 23, 2025}
Taylor diagrams are used to showcase the performance of multiple models against one another. This is done through a polar-style plot that intergrates three metrics: standard deviation (STD), centered-root-mean-squared-deviation (CRMSD) and the correlation coefficient (R2).

For most of the models, two out of three of the above are considered. Our goal here is to introduce the missing metrics and generate the Taylor Diagrams using the SkillMetrics or the CDAT library.

[pantoniad]

As per commit 86c7273, all of the metrics required to generate the Taylor Diagrams have been introduced.

Problem is now:

• When training, each model splits the dataset separately from the others, meaning that all models u+might be using different parts of the dataset for the training and validation.
• This leads to minor differences in the STD of the polynomial and GB regressors, but the difference with the ANN is huge.
• One possible way to test whether the split is the problem, is to have all of the models trained at once with the same part of the dataset and see the values retrieved for the metrics.
• The same happens for the CRMSD as well
• I cannot, yet, explain the huge difference between (nearly half) the std of the other two models and the ANN. But, as both the STD and the CRMSD have been implemented on the commit mentioned above, I have done something wrong in the code.

Regarding the Taylor Diagram:

• Some tests have been made to understand how the SkillMetrics library works. I am currently thinking of creating a wrapper method in the data_plotting class that handles the generation of these diagrams.
• Based on my above comments, I am also thinking of using Taylor Diagrams to plot how the different parameters used for the complexity plots are depicted on a Taylor diagram. Meaning that we no-longer care (for now) about comparing the performance of the three models with each other, rather we want to use Taylor diagrams to verify that the model we propose is, indeed, the best it can.

In general, I will have to look into the above problem first and check why the STD of the data used for the training and the testing of the ANN is so small (Pol, GBR: ~0.35, ANN: ~0.08 for testing on Idle)

[pantoniad]

On the above comment about the values of the STD and the CRMSD of the ANN being too small, I checked to see whether the batch file of the dataloader affects the results.

Idle conditions
For batch size = 30: Test CRMSD: 0.083, Validation data STD: 0.166,
For batch size = 40: Test CRMSD: 0.038, Validation data STD: 0.076
For batch size = 20: Test CRMSD: 0.0358, Validation data STD: 0.067
For batch size = 60: Test CRMSD: 0.2825, Validation data STD: 0.33

For reference:
Idle conditions, Polynomial Regressor,
Batch size = 30: Test CRMSD: 0.24, Validation data STD: 0.33

It turns out that the batch size was the one affecting the values of both STD and CRMSD (for the STD it was expected because its values is strictly based on the size of the examined dataset, the same can be said for CRMSD as well because of the consideration of 1/N where N is the number of datapoints considered).

Increasing the batch size, though, does not allow the algorithm to train on sub-sets of the training data at each iteration, helping reduce overfiting and improve generalisation.

Will keep batch size = 30 for now and test the hypothesis that all models should be trained on the same part of the dataset first.

[pantoniad]

As per commit 41a3612, I have identified multiple mishaps in the calculation of the metrics of the ANN model.

Their correction led to the recorded results being much better now. As stated in the commit, this did not affect the training or validation procedure of the model, as it was only exposed to the rmse value from the batch used and not from the averaged one. Plots and saved metrics show better results now that are closer to the ones given by other models.

Will skip the step that includes the comparison between models for now and will move on with the plotting of the Taylor diagrams.

[pantoniad]

For the generation of the complexity plots, I have devised the workflow below:

• The data generated for the complexity plots are going to be used as the basis for the inter-model comparison.
• For this, saving them to the right folder should be taken care of first,
• Then, a new execution and main function will be set up to handle the plotting per operating point,
• The plotting itself, will be done through a static method in the data_plotting class

The steps that need to be made, are:

1. Saving the results of the complexity plots,
2. Setting up the wrapper function in the data_plotting class and testing it. We want to follow examples 10 and 14 from the guide
3. A main and execution file should be set up to handle data collection and allocation per operating point

[pantoniad]

e0aee96

As per the last commit on the v0.1 branch, the first and the second points were successfully introduced. Some problems were encountered, as stated in the commit.