Iterators and Iterables simplified

Its very common to arise confusions on iterables and iterators in python . When it comes to apply on your code you will end up using the normal iterables as you are not getting the grip of Iterators

Whats Iterables ?

Iterables are considered as an object that can be looped over using for loop eg : List [1,33,55] tuple (33, 44, “hello”) dict {“name” : ”Jo”} etc..

we can iterate through them using a for loop

lst = [1,33,55]
for i in lst:
  print(i)

#prints
1
33
55

A simple for loop that traverse through the list , But under the hood something else is happening .

when you loop through an object it internally it executes like below

_it = iter(lst)          # creates a NEW iterator
while True:
    try:
        x = next(_it)    # calls __next__()
        print(x)
    except StopIteration:
        break

now we can see how the iteration is done !

1. creates an iterator _it = iter(lst)
2. starts an infinite loop while True:
3. calls the next function x = next(_it) starting from first element traverse trough every element
4. Stops iteration if no next element found. and breaks the loop
5. it actually raises an exception but for loop handles it using break . so we will not see the exception externally

Here we can achieve the iteration . But then what’s the role of an iterator ? while we can do the traversal with a for loop ?

Iterator is an object that can be traversed through an iterable and become exhausted after first traversal. Unlike the list(or any othe iterable which can be looped through many times) An iterator has __iter__() (returns itself) and __next__()function that returns next element

which is create an iterator using iter keyword as : lst = [1,22] iterate_object = iter(lst)

now we have an iterator object (iterate_object)

as stated before itearator has next() method so we can call it by one by one next(iterate_object) #gives 1 next(iterate_object) # gives 2 next(iterate_object) #gives stopIteration Exception

One key difference here is next is internally called by python for iterables with the help of for loop. But in case of iterator object we can call on demand whenever we need And we can loop through an iterable many times but for iterator object once its completed all object it will be exhausted and cannot be looped again .

Let’s check when the iterator object exhauseted and how it differ from a list

consider looping through an iterable (List here), and an iterator

lst = [1, 2, 3]

for x in lst:
    print(x)

for x in lst:
     print(x)   #here looping can be done again , no errros

But for an iterator object

it = iter([1, 2, 3])

for x in it:
    print(x)

for x in it:
    print(x)   #You won't get any output as the iteration already completed
               #above you won't get an exception as for loop manages it

Things to note: When iterating through list multiple times. for every new iteration a new iterator is created under the hood.

But for iterator object . it can only be looped once. or you need to create another iterator object

So when is iterator is usefull ?

1. Iterator is usefull when the iterable is not defined yet .

consider a list defined as lst = list(range(10_000_000_000)) the memory is used instatly but using it = iter(range(10_000_000_000)) for x in it: process(x)

here an iterator object is created but the value is generated one by one so huge memory usage difference her

2. Iterator is usefull for custom iterating logic we can create an iterable (list) using list coprehension like lst = [i for i in range (20) if i % 2 ==0] a custom logic is applied here and the complete list uses the memory

consider a custom iterator below that does the same logic

class MyIterator:
    def __init__(self, limit):
        self.limit = limit
        self.current = 0

    def __iter__(self):
        return self

    def __next__(self):
        while self.current < self.limit:
            val = self.current
            self.current += 1
            if val % 2 == 0:
                return val
        raise StopIteration

in the above iterator if the limit is huge (10000000) the iterator object does not create all the elements at once . only generated when we looping through one by one it = MyIterator(10000000)

Benifits . you can have a clean Iterator object to iterate with your custom logic. . the whole data aren’t generated once (helps when handling huge data) . No chance of repeating the loop by any chance