Programming

Full Name Parser With Python

How to clean and tide database fields of full names like names and surnames

Image by Predictive Hacks

[embed]Join Medium with my referral link - George Pipis Read every story from George Pipis (and thousands of other writers on Medium). Your membership fee directly supports…jorgepit-14189.medium.com

Many companies and individuals hold a database of names and there is a need to join them with one or more other databases or just to clean them by removing the duplicates. In most cases, their approach in this kind of task is to apply fuzzy matching using Levenshtein distance (see here). Usually, the fuzzy matching fails because every userbase has a different approach of storing and representing the data. For example, the name “George Pipis”, can be written as “George P. Pipis”, “Mr. George Prokopis Pipis”, “Dr. George Pipis Jr.”, “Pipis, George” etc. Clearly, in this case, the “fuzzy matching” does not work since we are not talking about a mi-spelling of a couple of characters.

Python Library nameparser

The python library called nameparser, gives us the chance to split the “Full Name” into “title”, “first”, “middle”, “last”, “suffix” and “nickname. For example the name “Dr. Juan Q. Xavier de la Vega III (Doc Vega)” can be decomposed as follows:

from nameparser import HumanName

name = HumanName("Dr. Juan Q. Xavier de la Vega III (Doc Vega)")

name

And we get:

<HumanName : [
    title: 'Dr.'
    first: 'Juan'
    middle: 'Q. Xavier'
    last: 'de la Vega'
    suffix: 'III'
    nickname: 'Doc Vega'
]>

Join Databases on Full Names

A good approach for joining the databases on “Full Names”, would be:

1. Split them into parts.
2. Convert them to lower case.
3. Remove the periods “.” symbols like (Dr. can be Dr).
4. The following example shows how we can easily get a pandas dataframe with a column with the “Full Name” and to create another 5 columns based on name parts.

from nameparser import HumanName

import pandas as pd

df = pd.DataFrame({'Name': ["George P. Pipis", "George Pipis", "Mr. George Prokopios Pipis", "Dr. George Pipis Jr.",

"Pipis, George", "Dr. Juan Q. Xavier de la Vega III (Doc Vega)"]})

df["title"] = df["Name"].apply(lambda x: HumanName(x).title)

df["first"] = df["Name"].apply(lambda x: HumanName(x).first)

df["middle"] = df["Name"].apply(lambda x: HumanName(x).middle)

df["last"] = df["Name"].apply(lambda x: HumanName(x).last)

df["suffix"] = df["Name"].apply(lambda x: HumanName(x).suffix)

df["nickname"] = df["Name"].apply(lambda x: HumanName(x).nickname)

df.drop('Name', axis=1)

This means that we can define some rules for joining or to give a priority like:

1. First match the exact match of the 5 parts.
2. Then the exact match of the “first”, “middle” ,”last”.
3. Then the exact match of the “first”, the first character of “middle”, “last”.
4. Then the exact match of the “first” and “last”.

Sometimes we will need also to apply fuzzy matching after the name parser. You get an idea of the concept of Text Distances and Fuzzy Joins.

Originally posted by Predictive Hacks