Add automated interpretability metrics

[alan-cooney]

Add automated interpretability metrics (using either a large local model, or e.g. GPT) to the validation stage of the pipeline.

[StatKumar]

@alan-cooney can I work on this issue? or any other issue you need help with.

[alan-cooney]

@StatKumar that would be great! Btw feel free to slack me if you have any questions on the open source mech interp group

As a quick starting thing - I think it'll want to extend AbstractValidationMetric and you may also want to update that abstract class a bit

[StatKumar]

Thank you @alan-cooney

[HoagyC]

@StatKumar are you still working on this? I've got some previous code from https://github.com/hoagyc/sparse_coding that I'd like to clean up and add as a metric.

[HoagyC]

Thinking about this a bit further:

• It shouldn't be a validation metric - it costs quite significant money to run (I think ~5p per feature interp at current prices, haven't checked recently) so you don't want to run on unfinished runs often.
• There's nothing that's specifically sparse autoencoder-related about doing automatic interpretability - all you need to do is define some function of the activations and evaluate it across a big range of inputs. I think having an easy-to-use version is good but might not make sense here. It might make more sense in TransformerLens?
• The way I did this before depended on https://github.com/openai/automated-interpretability which is not a package, just a normal git repo which might not be in line with something that wants to be a mature package? Maybe if doing this it would go into a separate part of the poe dependencies, like [dev, demos]?