Lightweight Language Detection using Langdetect

Language detection (Image generated by Gemini AI)

I wrote this article based on my experience.

I was working with text in many different languages and needed a way to automatically identify the language used.

The problem is that each text is quite long, and there were thousands of records, so performance was a major crucial.

That’s when I discovered langdetect, a lightweight Python library also used by Google Translate for language detection.

You can install the langdetect library using this command:

pip install langdetect

Take a closer look, the following script detects the language of each text sample and prints both the result and the time it took to process:

[embed]

Here is the result:

The results were quite accurate, and most detections were completed in under 3 ms. The first run took slightly longer due to the library’s initial loading.

To be more certain, I tested it on a larger dataset: 7k news articles, each averaging around 2k characters.

I can’t share the data due to work-related confidentiality. Here is the result:

It took 23.9 seconds to process all 7k records, around 3.08 ms per article. Quite fast, right?

Keep in mind that these results can vary based on hardware specifications and don’t exclude the initial load time.

Despite its great performance on long texts, langdetect tends to struggle with short texts. It may misidentify the language due to insufficient linguistic features.

I recommend using it only when the input text is long enough to provide reliable linguistic signals.

Thanks for reading! I hope this helped you understand how langdetect performs in real-world scenarios.