Building a Robust Tokenizer

In the previous part, creating a basic tokenizer, we created a very basic sentence tokenizer. Unfortunately, it had several pitfalls. So we…