[Question] how hard is it to implement HiddenState KD?

[mohamad-tohidi]

Your Question

thank you very much for the good project.
i read the documentations, and i saw that for distillation methods, you have already implemented the OPD. which is very nice.

i was wondering how hard is it to implement the HiddenState KD method.
i understand that the feature requests are out of your current scope and that is why im asking not to build this feature, but how hard is it, and what steps would it take for me to implement this.

What I've Tried

• i read the documentation pages.

Environment (if relevant)

• vime version:
• Python version:
• PyTorch version:
• CUDA/ROCm version:
• GPU type and count:
• OS:

Additional Context

https://github.com/arcee-ai/distillKit
this project has already implemented the asked feature, but i wanted to use vime.

Pre-submission Checklist

• I have read the CONTRIBUTING.md and understand the collaboration scope.
• I have read the documentation and FAQ and my question is not answered there.
• I have searched for existing issues and my question has not been asked before.