[Feature Request]: Trans-LoRA

[Katehuuh]

https://arxiv.org/pdf/2405.17258

Trans-LoRA: towards data-free Transferable Parameter Efficient Finetuning

Trans-LoRA can always apply LoRA across base models of any transformer architecture or of any size (e.g., from Gemma-7b to LLama-2-13B).

This would be extremely valuable if implemented, as it would let LoRA interfere with any new monthly SOTA model without new LoRA training.