I Built 3 Tokenizers From Scratch (And Finally Understand How LLMs Read Your Prompt)

I had heard the word a hundred times. I even used tokenizer.encode() without thinking about it. But this week, I sat down and build three…