paired linreg is not quite linreg

[NickDST]

It is two linear transformations instead of just one, the one below would have 8 Million parameters

[alam-shahul]

So as per our discussion: it is just effectively a single linear layer, but implemented as two matrices multiplied together. This is very inefficient, training-wise, but maybe not a big issue...? There are no non-linearities in between the layers, after all.

I guess technically speaking we should still fix this, since it's just weird and affects our transformer-based runs as well.