Monkey Patching: Quickly Fixing Bugs in Python Libraries

Monkey Patching: Quickly Fixing Bugs in Python Libraries

About eight months ago, I was working on a project that involved uploading and processing Excel files using Pandas. Although Pandas provided robust functionality, I found that it became increasingly slow and resource-intensive when handling larger datasets. To optimize performance, I decided to switch to Polars — a blazingly fast DataFrame library in Python known for its efficiency in handling large data volumes.

Encountering the Bug

While transitioning to Polars, I encountered an unexpected issue: Polars was unintentionally dropping empty rows when reading Excel files. This differed significantly from Pandas’ behavior, which preserved these empty rows, potentially causing data inconsistencies and impacting the accuracy of analysis.

Diving into Polars’ Codebase

To address this issue, I explored the Polars codebase. Its modular design made it easier to pinpoint the source of the problem: a helper function called _drop_null_data. This function was aggressively dropping rows containing null values, inadvertently removing intentionally empty rows that should have remained intact.

Quick Solution: Monkey Patching

Given the urgency of the issue, I opted for a quick and temporary fix using monkey patching — a technique in Python that dynamically modifies code at runtime.

Here’s the monkey patch I implemented:

import polars as pl

# Override the problematic function to simply return the input unmodified
def _drop_null_data(*args, **kwargs):
    return args[0]
# Apply monkey patch to Polars
pl.io.preadsheet.functions._drop_null_data = _drop_null_data

file_path = "file path of the excel file that needs to be loaded"
pl.read_excel(file_path, engine="calamine")

This patch immediately resolved the issue, ensuring Polars maintained the empty rows as expected and aligned its behavior with Pandas.

Contributing a Permanent Fix

Recognizing that monkey patching was only a short-term solution, I decided to contribute a permanent fix back to the Polars project. I created an issue on the Polars GitHub repository, clearly detailing the bug and seeking community input:

• GitHub Issue: Polars dropping empty rows while reading Excel

After discussions with Polars maintainers, I submitted a pull request (PR) containing a proper fix. The PR was reviewed, approved, and merged, permanently resolving the issue in the library:

• Merged PR: Fix Excel reader dropping empty rows

Lessons Learned

Through this experience, I gained valuable insights:

• Monkey patching offers a quick remedy but should ideally be temporary.
• Open-source contributions not only solve community problems but also foster deeper understanding and collaboration.
• Exploring library internals significantly enhances debugging skills and problem-solving capabilities.

Monkey patching was essential in quickly addressing the issue, but contributing a permanent solution to Polars ultimately provided a lasting impact for both myself and the broader Python community.