Weight tying in language models: when and why LLMs share embeddings

Most transformer models made the same architectural choice for years without questioning it. Then they stopped. This post looks at the data…