How I built a Language Detection Model Using Machine Learning

image from journaldunet.fr

So, I wanted to work on an exciting personal project, but I soon realized there were foundational concepts I needed to understand first. One of those was Natural Language Processing (NLP). So I decided to learn the concept and its applications. After juggling through theoretical materials, it was time to get the hands dirty by working on a project.

With several potential NLP projects available, I eventually settled for language detection. It felt like the perfect starting point. The goal was to classify text into different languages using traditional machine learning algorithms, rather than transformer models. I wanted to explore how far traditional machine learning algorithms could go in this case.

In this article I will walk you through the process, from initial challenge to model improvement and validation.

Dataset Overview and Exploration

I found a dataset with two columns: text, which contained sentences in different languages, and language, the corresponding labels.

Before jumping into model building, I explored the dataset and found out the dataset contains 22 different languages, each with 1,000 sentences, which makes 22,000 rows in total. This showed that the dataset was well-balanced, with each language equally represented. This was a great starting point, as I didn’t have to worry about class imbalance.

language distribution

Model Selection and Base Training

For my base models, I selected four models: Multinomial Naïve Bayes, Logistic Regression, Random Forest, Support Vector Classifier (SVC) which are popular algorithms for classification problems. Since the feature column i.e. text is not in numerical form and the ML algorithms can only work with inputs in numerical form, there is a need to convert the texts to numbers (vectors).

To process these texts, I used the Count Vectorizer method which is the implementation of bag of words from the scikit-learn library. I split the data into training and test sets, transformed the training feature and trained the models to check for their performances.

training set metrics

The Naïve Bayes model performed best with 98.40% accuracy on the training test and 94.38% accuracy on the test data, compared to the other algorithms. This shows the model is performing well and generalizing reasonably to the test data. While random forest classifier and logistic regression model seem to struggle with overfitting, support vector classifier show signs of underfitting with low accuracy on the train and test set. With this I decided to proceed with the Naïve Bayes model.

testing set metrics

To analyze where the naïve bayes model failed, I examined the confusion matrix and realized that languages with similar letters were sometimes misclassified. Also the model struggled to classify Chinese and Japanese as they were mostly classified as Arabic and English. This could be because the vectorizer ignored new words in the test data that were not present in the train data during transformation. This is called Out-of-Vocabulary error and is a major disadvantage of using bag of words method compared to word embeddings.

base model confusion matrix

Model improvement

To improve the model, I tried both the one-hot encoding and TF-IDF methods of vectorization. However, the results remained similar, suggesting that out-of-vocabulary words in test data was the main reason. Then I discovered that Count Vectorizer method uses ‘word’ as the default argument for its analyzer parameter, meaning it vectorizes text by splitting it into individual words. This made it much difficult for the model to handle unseen words in the test data. To address this I changed the analyzer argument to ‘char’. This means the vectorizer will now vectorize text by characters. To further improve detection, I added n-grams parameter, which considers sequences of characters instead of individual ones. I experimented with unigrams (single characters), bigrams (two-character sequences), trigrams (three-character sequences) and quadgrams (four-character sequences) and I settled with the combinations of unigrams and quadgrams (ngram_range = (1,4)) as this will allow the model to capture the structural patterns unique to each languages up to the fourth character and also balance the dimensionality of the features.

I retrained the model and checked for the metrics. The accuracy improved to 97.91% and the confusion matrix showed better performance on the Chinese and Japanese sentences.

improved model metrics

Validating the Model with New Data

To ensure the model worked beyond the dataset, I generated a new dataset with 132 new sentences across the 22 languages using AI, ensuring each language had representation. I also tested single text inputs, especially for Chinese, Japanese, and Portuguese, to see if the model could correctly identify them. The confusion matrix showed a perfect prediction and the single text inputs were correctly identified.

first test dataset metrics

To take things further, I generated another dataset with 152 new sentences across the languages, now with short sentences. I tested the model again and the model showed an accuracy of 91.56%. The confusion matrix also showed the languages that were misclassified by the model.

second test dataset metrics

I extracted a new dataframe to manually investigate where the problem lies, then I realized that short sentences have lower accuracy because the misclassified languages are very similar in pattern and they contain fewer distinguishing features. The model is able to predict those languages in long sentences and even short sentences many other times.

misclassified dataframe

Final Thoughts and Lessons Learned

Through this project, I realized the power of character-level vectorization in language detection. While word-based vectorization failed at unseen words, character n-grams captured more language patterns and character combinations, making them highly effective for multi-language classification.

Naïve Bayes, despite being a simple model, outperformed more complex algorithms due to its robustness against overfitting. It was also the fastest to train among the others. Sometimes, the best solutions are not the flashiest, they are just the ones that fit the problem best.

Traditional machine learning algorithms despite their limitations, proved to be lightweight, interpretable, and surprisingly effective. While Transformer-based models like mBERT could potentially handle out-of-vocabulary words better, they come at a higher computational cost. For this task, Naïve Bayes with character-level vectorization hit the best balance between accuracy and efficiency.

Would love to hear your thoughts! Have you worked on language detection or NLP projects before? Let’s discuss in the comments.

You can also try out the deployed version of the model here

Let’s Connect!

If you’re interested in exploring the full code, you can check out my GitHub repository here. I would love to connect with like-minded people, feel free to reach out on **LinkedIn or [X (Twitter)](https://x.com/max_d_don?s=21&t=5VLsXHChvNbCKP4acpuP3g)** .