Distilling a Teacher into a Student with Doc-to-LoRA Training

Large language models are mostly feed-forward weight. In a transformer, the attention layers do the routing, but the feed-forward layers…