🧠 Deciphering the Model’s Memory: Mutual Information and the Code-Length of Data

In Part 1, we introduced the idea that language models can “memorize” parts of their training data — sometimes harmlessly, sometimes in…