LLM Series — Tokenization Methods

The most common methods of tokenization range from word to byte , out of which some are not used that extensively now in language AI. Lets…