Effect of adding tau rescalings on emulator and sampler performance.

[Chris-Pedersen]

I keep coming back to this but we'd make more consistent progress on this issue by storing all the relevant plots in one place, so I'll do that in this issue here. The first 2 plots to look at would be the leave-one-out validation tests and a simple sampler run where we include tau rescalings. We can do this in 2 ways - either using the default hyperparameters optimised without tau rescalings, or re-optimising on the boosted training set.

[Chris-Pedersen]

To start off, for two randomly selected sims, (10 and 12), we remove the training points from that simulation from the training set and compare the truth in the simulation with the emulator prediction for 3 emulators:

1. No tau rescalings in the training set
2. With tau rescalings, using hyperparameters optimised on the training set without tau rescalings
3. With tau rescalings, re-optimising the hyperparameters on the full training set including tau rescalings

Sim 10 as test

Sim 12 as test

Whilst there is not much difference in emulator performance when the hyperparameters are kept the same, re-optimising them degrades performance significantly. The hyperparameters are:

No rescalings
sigma_lin=25.1
l_rbf=1.08
sigma_rbf=30.5
noise_var=7.1e-6

With tau rescalings, test sim 10
sigma_lin=20.3
l_rbf=0.45
sigma_rbf=4.8
noise_var=9.5e-15

With tau rescalings, test sim 12
sigma_lin=18.9
l_rbf=0.44
sigma_rbf=5.9
noise_var=1.8e-25

Assuming these two test cases are representative of the effect of adding tau rescalings, they seem to roughly halve the correlation length in the RBF kernel, and play havoc with the noise_var term. It's not immediately obvious to me why adding tau rescalings would have this effect on the optimised hyperparameters, @keirkwame do you have any thoughts?