Customizing Standard Python Packages

We use libraries like pandas, numpy more frequently. There are operations that are standard to day-to-day work and organization specific. We can always write procedures and call them to automate the repetitive tasks. However, the objective of the exercise was to check if the popular libraries can be modified with the help of LLMs to fit custom use cases. The experiment uses the original library as base and modifies it just enough to fit the use case.

Modifying Pandas to fit custom use cases

Cases Tested

1. CSV Read Time
2. Data Clean Up
3. Data Summarization

For testing we used a file with 514K rows and different data types.

Outcome

1. CSV Read Time

2. Data Clean Up Cleaner code syntax

#Original Pandas 

df = df.dropna(how='all')
df = df.dropna(axis=1, how='all')
for col in df.select_dtypes(include='integer').columns:
  df[col] = pd.to_numeric(df[col], downcast='integer')
df['Date'] = pd.to_datetime(df['Date'], format='%Y-%m-%d', errors='coerce')

#modified 

op.clean(op_df, cast_types=True, dates=True, date_columns=['Date'],
                     nulls=True, text=True, dedup=True)

3. Data Summarization

df.describe() + isnull().sum() + duplicated().sum() + memory_usage(deep=True).sum() vs op.summary(df)

Additional Features

One step data quality report

miss = op.missing_report(op_df)
cat  = op.categorical_summary(op_df)
num  = op.numeric_summary(op_df) 

The next step of the experiment is to build more custom Airflow packages as per Organization needs.

You can access the code here to test the Opandas . Please create a separate virtual env for testing .

https://codeberg.org/varunk/public.git