Python Iterators: __iter__ and __next__ Methods
Every time you write a `for` loop in Python, you're using iterators. They're the mechanism that powers Python's iteration protocol, enabling you to traverse sequences, streams, and custom data...
Key Insights
- Iterators implement the
__iter__()and__next__()methods to enable sequential access to elements without loading everything into memory at once - Understanding the difference between iterables (objects that return an iterator) and iterators (objects that track iteration state) is crucial for avoiding exhaustion bugs
- Custom iterators are best for complex stateful iteration logic, while generators are simpler for most use cases—know when to use each
Introduction to Iterators
Every time you write a for loop in Python, you’re using iterators. They’re the mechanism that powers Python’s iteration protocol, enabling you to traverse sequences, streams, and custom data structures with consistent syntax. More importantly, iterators allow you to process data sequentially without loading entire datasets into memory—a critical feature for handling large files, database results, or infinite sequences.
At its core, an iterator is an object that implements two methods: __iter__() and __next__(). Together, these methods form the iterator protocol, a contract that allows Python’s iteration machinery to work with your objects.
Here’s what happens behind the scenes when you iterate over a list:
# High-level iteration
numbers = [1, 2, 3, 4, 5]
for num in numbers:
print(num)
# What Python actually does
numbers = [1, 2, 3, 4, 5]
iterator = iter(numbers) # Calls numbers.__iter__()
while True:
try:
num = next(iterator) # Calls iterator.__next__()
print(num)
except StopIteration:
break
The iter() function calls the object’s __iter__() method to get an iterator, and next() repeatedly calls __next__() until a StopIteration exception signals the end of the sequence.
The Iterator Protocol: iter and next
To create an iterator, your class must implement two methods:
__iter__(): Returns the iterator object itself (usuallyself)__next__(): Returns the next value in the sequence or raisesStopIterationwhen exhausted
Here’s a simple counter iterator that demonstrates the protocol:
class Counter:
def __init__(self, start, end):
self.current = start
self.end = end
def __iter__(self):
return self
def __next__(self):
if self.current > self.end:
raise StopIteration
value = self.current
self.current += 1
return value
# Usage
counter = Counter(1, 5)
for num in counter:
print(num) # Prints 1, 2, 3, 4, 5
The __iter__() method returns self because Counter is both an iterable and an iterator. The __next__() method manages the iteration state, returning values until the condition is met, then raising StopIteration to signal completion.
Building a Custom Iterator Class
Let’s build something more practical: an iterator that traverses a sequence in reverse without creating a reversed copy in memory.
class ReverseIterator:
def __init__(self, sequence):
self.sequence = sequence
self.index = len(sequence)
def __iter__(self):
return self
def __next__(self):
if self.index == 0:
raise StopIteration
self.index -= 1
return self.sequence[self.index]
# Usage
words = ['apple', 'banana', 'cherry', 'date']
for word in ReverseIterator(words):
print(word) # Prints: date, cherry, banana, apple
This iterator maintains an index that starts at the sequence length and decrements with each call to __next__(). When the index reaches zero, iteration stops. This approach uses constant memory regardless of sequence size, unlike reversed() which creates a new iterator object or slicing which creates a new list.
Iterators vs. Iterables
This distinction trips up many Python developers: an iterable is any object that can return an iterator (implements __iter__()), while an iterator is an object that represents a stream of data (implements both __iter__() and __next__()).
Lists, tuples, and strings are iterables but not iterators. They can be iterated over multiple times because each call to iter() returns a fresh iterator. Iterators, on the other hand, are exhaustible—once consumed, they’re done.
# Iterable that returns a new iterator each time
class NumberCollection:
def __init__(self, numbers):
self.numbers = numbers
def __iter__(self):
return NumberIterator(self.numbers)
class NumberIterator:
def __init__(self, numbers):
self.numbers = numbers
self.index = 0
def __iter__(self):
return self
def __next__(self):
if self.index >= len(self.numbers):
raise StopIteration
value = self.numbers[self.index]
self.index += 1
return value
# This works multiple times
collection = NumberCollection([1, 2, 3])
print(list(collection)) # [1, 2, 3]
print(list(collection)) # [1, 2, 3] - works again!
# Compare with an iterator
iterator = iter([1, 2, 3])
print(list(iterator)) # [1, 2, 3]
print(list(iterator)) # [] - exhausted!
The key difference: NumberCollection creates a new NumberIterator instance each time __iter__() is called, preserving the original data for repeated iteration.
Practical Use Cases
Custom iterators shine when you need stateful iteration logic or memory-efficient processing of large datasets. Here are two practical examples:
Chunked File Reader: Process large files without loading them entirely into memory.
class ChunkedFileReader:
def __init__(self, filename, chunk_size=1024):
self.filename = filename
self.chunk_size = chunk_size
self.file = None
def __iter__(self):
self.file = open(self.filename, 'r')
return self
def __next__(self):
chunk = self.file.read(self.chunk_size)
if not chunk:
self.file.close()
raise StopIteration
return chunk
# Process a large log file in 4KB chunks
for chunk in ChunkedFileReader('large_log.txt', 4096):
process_chunk(chunk)
Fibonacci Iterator: Generate infinite sequences on-demand.
class FibonacciIterator:
def __init__(self, max_value=None):
self.max_value = max_value
self.a, self.b = 0, 1
def __iter__(self):
return self
def __next__(self):
if self.max_value and self.a > self.max_value:
raise StopIteration
current = self.a
self.a, self.b = self.b, self.a + self.b
return current
# Generate Fibonacci numbers up to 100
for num in FibonacciIterator(100):
print(num) # 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89
These iterators handle state management internally and produce values lazily, making them ideal for streaming data processing, pagination APIs, or mathematical sequences.
Common Pitfalls and Best Practices
Iterator Exhaustion: The most common mistake is trying to reuse an exhausted iterator.
numbers = iter([1, 2, 3])
print(sum(numbers)) # 6
print(sum(numbers)) # 0 - iterator is exhausted!
# Solution: Use itertools.tee() to create independent iterators
from itertools import tee
original = iter([1, 2, 3])
iter1, iter2 = tee(original, 2)
print(sum(iter1)) # 6
print(sum(iter2)) # 6 - works because it's independent
When to Use Generators Instead: For most simple iteration needs, generator functions are cleaner:
# Iterator class (verbose)
class SquaresIterator:
def __init__(self, n):
self.n = n
self.current = 0
def __iter__(self):
return self
def __next__(self):
if self.current >= self.n:
raise StopIteration
result = self.current ** 2
self.current += 1
return result
# Generator function (concise)
def squares(n):
for i in range(n):
yield i ** 2
# Both work the same way
for sq in SquaresIterator(5):
print(sq)
for sq in squares(5):
print(sq)
Use custom iterator classes when you need:
- Complex state management across multiple methods
- Objects that are both containers and iterators
- Fine-grained control over iteration behavior
- Integration with existing class hierarchies
Use generators when you need:
- Simple sequential value generation
- One-off iteration logic
- Cleaner, more readable code
Resource Management: Always clean up resources in iterators that open files or connections. Consider implementing __del__() or using context managers:
class SafeFileIterator:
def __init__(self, filename):
self.filename = filename
self.file = None
def __enter__(self):
return self
def __exit__(self, exc_type, exc_val, exc_tb):
if self.file:
self.file.close()
def __iter__(self):
self.file = open(self.filename, 'r')
return self
def __next__(self):
line = self.file.readline()
if not line:
raise StopIteration
return line.strip()
# Use with context manager
with SafeFileIterator('data.txt') as iterator:
for line in iterator:
print(line)
Conclusion
Understanding iterators is fundamental to writing efficient Python code. The iterator protocol—built on __iter__() and __next__()—provides a consistent interface for sequential data access while enabling memory-efficient processing of large or infinite datasets.
Implement custom iterator classes when you need complex state management or want to integrate iteration behavior into your object model. For simpler cases, reach for generator functions or the excellent itertools module. Always remember the distinction between iterables and iterators to avoid exhaustion bugs, and clean up resources properly when your iterators manage external connections or files.
Master these concepts, and you’ll write more efficient, Pythonic code that handles data streams elegantly and scales to handle datasets of any size.