“Duplicate Entries” — From Databases to Voter Lists

Duplicate entries are one of the biggest challenges in the data world. Any system that’s accessed by multiple channels without proper integration, validation, or constraints is bound to create duplicates sooner or later.

Interestingly, this isn’t just a data engineering problem anymore it has now reached India’s voter lists! Rahul Gandhi has recently started highlighting this issue (almost like a client’s technical manager pointing out a long-standing bug 😄).

And as always, the clients (common people) are shocked, but the developers (Election Commission) just nod quietly “Yeah, it’s a known issue”. They’ll probably mark it as a P3/P4 ticket and assign it to someone who doesn’t even know where to start fixing duplicates.

Because here, the key column is unique… but the attributes are duplicated! 😅

Back in 2016, when I was actively involved in NGO activities, our small team worked on something similar. We downloaded hundreds of voter lists street-wise PDFs from the Election Commission website for the Thoraipakkam area. Using a simple Python script, we extracted text from the PDFs and ran a duplicate check.

Guess what we found? A person named Yasodha appeared nine times on the voter list for Thoraipakkam!

We brought this anomaly to the public through a press release, and the story was published across several news platforms (link). That was actually the first time my name appeared in print media a proud moment for me!

Soon after, the Election Commission took two smart (and slightly defensive 😄) steps:

1. Added a CAPTCHA to block programmatic downloads.
2. Changed the PDFs into non-copyable image files, preventing text extraction.

That was the pre-AI era, and our efforts were effectively stopped. But it felt good to have raised awareness about such an important data issue.

Now, years later, Rahul Gandhi and his team with far more technical power and data capabilities are doing a similar exercise at scale, uncovering anomalies and duplicates in voter lists across India.

It’s high time our electoral systems become more data-driven, integrated, and constraint-enabled because these are not just data quality issues anymore; they impact democracy itself.

Data issues are everywhere :)