Iterator vs generator in Python

Working with large datasets and storing everything in memory isn’t always efficient and practical. This is where iterators and generators come in, allowing you to process data efficiently. This article will explore these concepts and show how they work.

Python provides many powerful packages to handle data efficiently. Two of its key features, iterators and generators, allow us to process sequences memory-efficiently using the lazy evaluation concept. In this article, we’ll explore:

• What iterable, iterators, and generators are?
• What and how does lazy evaluation work and why it’s useful?
• Key differences between iterators and generators?
• When should each approach be used in the real world?

Understanding iterator in Python

What is an iteration?

Iteration is a fundamental concept in Python. it refers to the process of accessing elements one by one from a collection, whether you’re looping through a list, reading lines from a file, or processing incoming data streams.

However, not all iterable objects are iterators. To understand this better, we need to distinguish between iterable and iterators.

What is an iterable?

An iterable in Python is an object that can be looped over using a for loop. It contains multiple elements, such as those in a list, tuple, or string. An iterable does not keep track of its position during iteration. Instead, it has a special method, iter(), which returns an iterator that allows sequential access to its elements.

Examples of Iterables in Python:

• Lists → [1, 2, 3]
• Tuples → (10, 20, 30)
• Dictionaries → {“name”: “Alice”, “age”: 25}
• Strings → “hello”

💡 Important: An iterable itself is NOT an iterator. It only provides an iterator when iter(iterable) is called.

how does an iterable work?

When we use a for loop, Python automatically calls the iter() function on the iterable. This function returns an iterator, which is responsible for fetching elements one by one. The loop continues until all elements have been accessed, making iteration seamless and efficient.

my_list = [2, 4, 6, 8, 10, 12]
for item in my_list:
      print(item)

image from code360

What is the iterator?

An iterator in Python is an object that retrieves elements one at a time while keeping track of its current position in the sequence. It does not store all elements in memory but generates them as needed. An iterator must implement two methods: iter(), which returns the iterator itself, and next(), which returns the next element in the sequence.

Each call to next() advances the iterator, and when no more elements are available, it raises a StopIteration exception. Since an iterator is consumed as it progresses, it cannot be reset or reused without creating a new iterator from the original iterable.

How does the iterator work?

An iterator is created from an iterable (such as a list) using the iter() function. Once the iterator is created, we can find the next item in the sequence using the next() function. Each time the next() is called, the iterator returns the next element in the sequence. When there are no more items to return, the iterator raises a StopIteration exception to signal the end of the iteration, which stops the loop.

my_list = [24, 78, 'coding', 'is', '<3']
iterator = iter(my_list)  # Create an iterator

print(next(iterator))  # Output: 1
print(next(iterator))  # Output: 2

In this example, the next() function fetches elements from my_list one by one. Once the iterator reaches the end, the StopIteration exception is raised.

image from code360

Understanding Generator in Python

What is lazy evaluation?

Lazy evaluation is a programming technique where values are computed only when needed, rather than being computed upfront. This means that instead of calculating all values at once and storing them in memory, the program computes values one at a time as they are requested

How it works?

Here’s a breakdown of how it works:

• One value at a time: Instead of calculating and storing every value in advance, the object generates values only when requested, saving memory and computation time.
• Pause until next request: After producing a value, the function pauses its execution until the next value is needed. This means it doesn’t keep the entire dataset in memory, which is especially useful for large datasets or streams of data.
• Efficiency with large datasets: Because it doesn’t load everything into memory upfront, lazy evaluation allows iterators and generators to handle large datasets efficiently without running into memory limitations.

This approach is particularly useful when dealing with long-running processes or large amounts of data that don’t need to be fully loaded into memory at once.

What is the generator?

A generator in Python is a special type of iterator defined using a function with the yield keyword. It produces values one at a time and maintains its state automatically between iterations. Generators use lazy evaluation, meaning that they compute and return values only when they are needed, rather than calculating all values upfront. This makes them memory-efficient and ideal for working with large or infinite datasets.

How it Works:

• Calling the generator function: When you call a generator function, it doesn’t execute immediately but returns a generator object.
• First call to next(): The function starts executing and runs until it encounters a yield statement, at which point it pauses and returns the value to the caller.
• Subsequent calls to next(): The function resumes execution from where it left off after the last yield, continuing until the next yield is encountered.
• End of iteration: Once all values have been produced, or when the function finishes, a StopIteration exception is raised.

def count_up_to(max):
    count = 1
    while count <= max:
        yield count  # Pauses and returns current count
        count += 1

# Create a generator object
counter = count_up_to(3)

# Call next() to get values one at a time
print(next(counter))  # Output: 1
print(next(counter))  # Output: 2
print(next(counter))  # Output: 3
# Calling next(counter) now raises StopIterationprint(next(iterator))  
print(next(iterator))  # Raises StopIteration (end of list)

Summary: Comparing iterator and generator

Conclusion

We explored the differences between iterators and generators, understanding how they work and their advantages. This knowledge helps us choose the best approach to optimize memory usage and improve the efficiency of our programs.