tqdm() behind the scenes

A custom implementation of tqdm from scratch

*In this article, we will study:

• What is tqdm and its implementation to understand how it displays a progress bar
• What happens internally when an iterable is wrapped inside tqdm(…)*

For those new to tqdm, it’s an open-source Python library that provides a fast, extensible progress bar for loops and iterable operations. It helps track the progress of long-running tasks intuitively and visually appealingly.

For the ones already using tqdm, it’s interesting to know that tqdm is derived from “taqaddum” (an Arabic word for “progress”)

To use it, import the library and wrap it over an iterable:

Let’s answer some of the questions that got me wondering years ago when I first came across this library:

(1) How does it display this cool progress bar

An example implementation using “f-string” would be:

There are three core things that you need to know about the gist above:

• f-string is used above to create a constant spaced [and]using “:<100”that contains the “block” characters (█) representing the progress.
• Carriage return “\r” is used to bring back the cursor to the beginning of the line — so that each iteration of the for loop writes on top of existing printed progress bar.
• sys.stdout.flush() to make sure that the output is printed in the terminal immediately, and is not buffered.

(2) How does it know for how long it has to run

For this, we need to understand what happens when we do tqdm( … ) :

By looking at it, can we infer if tqdm is a class or a function?

As per the naming convention, shouldn’t it be a function since classes always start with a Capital letter? Well, not always — tqdm is an example.

The naming convention is just a “convention” — not a “mandate”. So the developers of tqdm probably decided to name it lowercase so that it gives you a functional vibe. Here is the official tqdm class.

Now, since it’s a class, there would be an initializer, data, and member functions. To understand these components, let’s build them step by step.

(1) First, when we do tqdm( ), an object of the tqdm class is initialized with the wrapped iterable.

(2) Second, focus on the “in” part — hinting that tqdm(…) it returns an iterator, and by definition, each iterator has to have a __iter__ dunder method to it (or vice versa)

The __iter__ method fetches the next element from the iterable and yields it.

(3) And finally comes the progress itself. To understand what’s required to track and update the progress, let’s revisit the main components of a progress bar:

So, all we need is some way to:

1. get total number of elements in the given iterable
2. identify the number of elements processed so far
3. display the progress bar

These components are implemented as follows:

That’s it. We just implemented our custom tqdm from scratch.

Next time you use tqdm, you’ll know exactly what’s happening under the hood!

🚀 May your loops always run with progress! :)

Originally Published at Intuitive Shorts:

[embed]#16 | tqdm() behind the scenes A custom implementation of tqdm from scratchintuitiveshorts.substack.com

Follow Intuitive Shorts (a Free Substack newsletter), to read quick and intuitive summaries of ML/NLP/DS concepts. Best Part — it just takes 1–2 minutes