Polyglean: A Text Cleaning Library That Works for Any Language

Built while preparing datasets for low-resource languages and fixing the silent failures left by English-first NLP tools.

Sourc: Image from Author

Most text cleaning tools are quietly English-first.

They work beautifully on Latin-script data. Then you hand them Yoruba, Hausa, Arabic, or Devanagari text, and they fail silently. No error. No warning. Just wrong output that looks right until you’re deep enough into your pipeline to notice.

I hit this wall while building TTS (text-to-speech) voice datasets for four Nigerian languages. The data came from multiple sources in multiple formats: Excel spreadsheets, scraped JSON, plain text, and OCR-extracted newspapers. Every language needed the same pipeline: load, clean, deduplicate, filter by word count, save — but the tools I reached for kept making the same wrong assumptions.

So I built Polyglean: a universal text-cleaning pipeline that works correctly for any Unicode script, any file format, and any language.

Github: https://github.com/dammmie-beep/polyglean

pip install polyglean

The Bugs That Motivated This

Bug 1: is_all_caps() that ignores your entire script

Filtering out all-caps rows is standard practice; they’re usually headers, section titles, or noise. The naive implementation looks like this:

# ❌ Broken for non-Latin scripts
def is_all_caps(text: str) -> bool:
    letters = re.sub(r"[^a-zA-ZÀ-ÿ]", "", text)
    return len(letters) > 0 and letters == letters.upper()

That regex strips out every character outside the Latin range. So when you pass in a Hausa sentence, for example, written in all-caps, the function sees an empty string and returns False

Bug 2: Deduplication that misses half your duplicates

Some languages uses tonal diacritical marks: ẹ́, ọ̀. The problem is that the same character can be stored in two different ways in Unicode:

• NFC (precomposed): ẹ́ as a single codepoint
• NFD (decomposed): e + combining dot below + combining acute — three separate codepoints

Both look identical on screen. A Python string equality check sees them as completely different. So the same sentence, copy-pasted from two different sources, survives your deduplication step as two separate rows. At scale, this creates thousands of false duplicates.

Bug 3: One pipeline per file format

Real NLP datasets don’t come in one format. Some of my data was in Excel, CSV, JSON, etc. Every new source meant writing another loader, another normaliser, another merge step, the same boilerplate, over and over.

The Fixes That Make Polyglean

Fix 1: Unicode Normalisation (NFC)

The diacritics problem has a one-line solution. Run it first, before anything else:

import unicodedata

def normalize_unicode(text: str) -> str:
    return unicodedata.normalize("NFC", text)

Fix 2: Unicode-Aware is_all_caps()

The fix is to stop asking “is this an ASCII letter?” and instead ask “what category of Unicode character is this?”

# ✅ Works for any script
import unicodedata

def is_all_caps(text: str) -> bool:
    cased = [ch for ch in text if unicodedata.category(ch) in ("Lu", "Ll", "Lt")]
    return len(cased) > 0 and all(unicodedata.category(ch) == "Lu" for ch in cased)

unicodedata.category() classifies every Unicode character using the standard Unicode property system:

Source: Image from Author

This works for Cyrillic, Greek, Armenian, and any script that has a case distinction. Scripts without case distinction (Arabic, CJK) are never flagged as all-caps, which is the correct behaviour.

Fix 3 — Multi-Format Source Loading

Polyglean handles all common dataset formats through a single unified source config:

source_config = [
    # Excel (shorthand tuple: path, column, label)
    ("bbc_news.xlsx", "sentence", "bbc"),

    # CSV with tab separator
    {"path": "data.tsv", "col": "text", "label": "web",
     "type": "csv", "sep": "\t"},

    # Parquet
    {"path": "corpus.parquet", "col": "sentence",
     "label": "hf", "type": "parquet"},

    # JSON array of objects
    {"path": "stories.json", "col": "story_text",
     "label": "stories", "type": "json_list"},

    # Plain text — one sentence per line
    {"path": "sentences.txt", "label": "misc", "type": "txt"},
]

Every entry gets loaded, labelled with its source, and merged into a single DataFrame. The source column survives all the way to the output; you always know where each sentence came from.

An Example Usage

Here is what a full language config looks like. One dict, one function call:

import polyglean

config = {
    "base_dir":       "/path/to/yoruba/data/",
    "output_dir":     "/path/to/output/",
    "source_config":  [
        ("bbc_yoruba.xlsx",   "sentence", "bbc"),
        ("bible_yoruba.xlsx", "sentence", "bible"),
        ("news_yoruba.xlsx",  "sentence", "news"),
    ],
    "separate_files": [],
    "extra_cleaners": [],
    "word_count_min": 5,
    "word_count_max": 20,
    "output_tts":   "yoruba_tts.xlsx",
    "output_other": "yoruba_other.xlsx",
}

polyglean.run_language("yoruba", config)

Running this produces two output files:

Source: Image from Author

The pipeline logs every step:

============================================================
  Language: YORUBA
============================================================

[1/4] Loading sources...
  Loaded  12,483 rows  ← bbc_yoruba.xlsx
  Loaded  94,221 rows  ← bible_yoruba.xlsx
  Loaded   8,374 rows  ← news_yoruba.xlsx
  Total rows loaded: 115,078

[2/4] Cleaning...
  Rows after clean + dedup: 89,432
  Word count range: 1–312
  In 5–20 word range: 61,204

[3/4] Saving merged files...
  Saved  61,204 rows → yoruba_tts.xlsx
  Saved  28,228 rows → yoruba_other.xlsx

What about the long sentences?

The rows in yoruba_other.xlsx aren't junk; there are many are long paragraphs containing perfectly good material that just needs splitting. Polyglean handles this too:

polyglean.split_other_sentences("yoruba", config)

This splits on sentence-boundary punctuation (. , ;), cleans each chunk, and saves yoruba_split_sentences.xlsx with word counts, using the same word_count_min/word_count_max from your config. No hardcoded magic numbers.

Adding Your Own Language

Adding a new language requires one config block and zero changes to the library itself. Here is a Swahili example with a custom cleaner to strip a dataset-specific prefix:

import re
from polyglean import LANGUAGES, run_language
from polyglean.cleaners import remove_urls

LANGUAGES["swahili"] = {
    "base_dir":       "/path/to/swahili/",
    "output_dir":     "/path/to/output/",
    "source_config":  [
        ("swahili_news.xlsx", "sentence", "news"),
        {"path": "web_data.csv", "col": "text",
         "label": "web", "type": "csv"},
    ],
    "separate_files": [],

    # Applied after the universal BASE_CLEANERS
    "extra_cleaners": [
        remove_urls,
        lambda text: re.sub(r"^Habari:\s*", "", text),
    ],

    "word_count_min": 5,
    "word_count_max": 20,
    "output_tts":   "swahili_tts.xlsx",
    "output_other": "swahili_other.xlsx",
}

run_language("swahili", LANGUAGES["swahili"])

The extra_cleaners list is the extension point for anything language-specific. The base pipeline: Unicode normalisation, control character removal, bracket stripping, whitespace collapsing, runs automatically for every language.

What’s Next

Polyglean v0.1.1 is functional and tested across Yoruba, Hausa, Igbo, and Nigerian Pidgin.

The roadmap:

• Tests and CI — a pytest suite and GitHub Actions workflow
• Non-space-segmented scripts — Languages that don’t use spaces between words; word count needs a character-based fallback or a pluggable tokeniser
• Sentence quality scoring — filter by perplexity or length distribution, not just word count
• YAML config files — define language configs in a file, not in code

Try It

pip install polyglean

GitHub: github.com/dammmie-beep/polyglean

Contributions are very welcome. whether that’s adding a language config, improving the sentence splitter, or filing a bug report. If you work with low-resource language data, I’d especially love to hear about formats and cleaning challenges I haven’t run into yet. Open an issue, open a PR, or just star the repo if it was useful.