Python - iter() and next() Functions

Key Insights

The iter() and next() functions are the foundation of Python’s iteration protocol, and understanding them unlocks powerful patterns for memory-efficient data processing.
The two-argument form of iter(callable, sentinel) is an underused feature that elegantly handles “read until condition” patterns.
Iterators are single-use by design—once exhausted, they’re done—which catches many developers off guard but enables significant memory optimizations.

Introduction to Iteration in Python

Every time you write a for loop in Python, you’re using the iterator protocol without thinking about it. The iter() and next() functions are the machinery that makes this possible, and understanding them transforms you from someone who uses Python to someone who thinks in Python.

The iterator protocol is simple: an object is iterable if it can return an iterator, and an iterator is an object that produces values one at a time until it’s exhausted. This lazy evaluation model is why you can iterate over a 10GB file without loading it into memory.

Most Python developers never call iter() or next() directly. That’s fine for basic work, but it limits what you can build. Once you understand these primitives, you can create custom data streams, implement pagination systems, and write memory-efficient processing pipelines.

Understanding the iter() Function

The iter() function converts an iterable into an iterator. The distinction matters: a list is iterable (you can iterate over it), but it’s not an iterator itself. When you call iter() on a list, you get a list iterator object.

numbers = [1, 2, 3, 4, 5]
iterator = iter(numbers)

print(type(numbers))    # <class 'list'>
print(type(iterator))   # <class 'list_iterator'>

The iterator maintains state—it knows where it is in the sequence. The original list doesn’t change; the iterator is a separate object that walks through it.

The Two-Argument Form

Here’s where iter() gets interesting. When you pass two arguments—a callable and a sentinel value—iter() creates an iterator that calls the function repeatedly until it returns the sentinel.

# Read lines from a file until we hit an empty line
with open('data.txt') as f:
    for line in iter(f.readline, '\n'):
        print(line.strip())

This pattern replaces awkward while loops:

# The old way (verbose and error-prone)
with open('data.txt') as f:
    while True:
        line = f.readline()
        if line == '\n':
            break
        print(line.strip())

The two-argument form shines when reading from sockets, processing queues, or any scenario where you’re pulling data until a termination condition:

import random

# Simulate rolling dice until you get a 6
def roll_dice():
    return random.randint(1, 6)

rolls = list(iter(roll_dice, 6))
print(f"Rolled {len(rolls)} times before getting a 6: {rolls}")

Understanding the next() Function

The next() function retrieves the next value from an iterator. Each call advances the iterator’s internal position.

colors = iter(['red', 'green', 'blue'])

print(next(colors))  # red
print(next(colors))  # green
print(next(colors))  # blue
print(next(colors))  # Raises StopIteration

When an iterator is exhausted, next() raises StopIteration. This is how Python knows when to exit a for loop. But in manual iteration, catching exceptions for flow control is ugly.

The Default Parameter

The second argument to next() provides a default value instead of raising an exception:

colors = iter(['red', 'green', 'blue'])

print(next(colors, 'no color'))  # red
print(next(colors, 'no color'))  # green
print(next(colors, 'no color'))  # blue
print(next(colors, 'no color'))  # no color (no exception)

This is cleaner than wrapping everything in try/except:

# Grab the first item or None
first_user = next(iter(users), None)

# Skip header row, get first data row
data_iter = iter(csv_rows)
next(data_iter)  # skip header
first_data = next(data_iter, None)

The Iterator Protocol Under the Hood

Python’s iteration is built on two dunder methods: __iter__() and __next__(). When you call iter(obj), Python calls obj.__iter__(). When you call next(iterator), Python calls iterator.__next__().

Here’s what a for loop actually does:

# This for loop...
for item in collection:
    process(item)

# ...is equivalent to this:
iterator = iter(collection)
while True:
    try:
        item = next(iterator)
    except StopIteration:
        break
    process(item)

You can verify this by manually iterating:

numbers = [10, 20, 30]
iterator = iter(numbers)

print(next(iterator))  # 10
print(next(iterator))  # 20

# Now use a for loop on the same iterator
for num in iterator:
    print(num)  # Only prints 30—the iterator remembers its position

Building Custom Iterators

Creating your own iterator means implementing __iter__() and __next__(). Here’s a countdown iterator:

class Countdown:
    def __init__(self, start):
        self.current = start
    
    def __iter__(self):
        return self
    
    def __next__(self):
        if self.current <= 0:
            raise StopIteration
        value = self.current
        self.current -= 1
        return value

# Usage
for num in Countdown(5):
    print(num)  # 5, 4, 3, 2, 1

For infinite sequences, never raise StopIteration:

class Fibonacci:
    def __init__(self):
        self.prev = 0
        self.curr = 1
    
    def __iter__(self):
        return self
    
    def __next__(self):
        value = self.curr
        self.prev, self.curr = self.curr, self.prev + self.curr
        return value

# Get first 10 Fibonacci numbers
fib = Fibonacci()
first_ten = [next(fib) for _ in range(10)]
print(first_ten)  # [1, 1, 2, 3, 5, 8, 13, 21, 34, 55]

Practical Use Cases

Chunked File Processing

When processing large files, loading everything into memory is a mistake. An iterator that yields chunks keeps memory usage constant:

class ChunkedFileReader:
    def __init__(self, filepath, chunk_size=1024):
        self.filepath = filepath
        self.chunk_size = chunk_size
        self.file = None
    
    def __iter__(self):
        self.file = open(self.filepath, 'rb')
        return self
    
    def __next__(self):
        chunk = self.file.read(self.chunk_size)
        if not chunk:
            self.file.close()
            raise StopIteration
        return chunk

# Process a 10GB file with constant memory
for chunk in ChunkedFileReader('massive_file.bin', chunk_size=8192):
    process_chunk(chunk)

Database Pagination

Fetching millions of rows at once kills your application. A pagination iterator fetches in batches:

class PaginatedQuery:
    def __init__(self, query_func, page_size=100):
        self.query_func = query_func
        self.page_size = page_size
        self.offset = 0
        self.current_page = []
        self.exhausted = False
    
    def __iter__(self):
        return self
    
    def __next__(self):
        if not self.current_page and not self.exhausted:
            self.current_page = self.query_func(
                limit=self.page_size, 
                offset=self.offset
            )
            self.offset += self.page_size
            if not self.current_page:
                self.exhausted = True
        
        if not self.current_page:
            raise StopIteration
        
        return self.current_page.pop(0)

# Usage
def fetch_users(limit, offset):
    return db.execute(f"SELECT * FROM users LIMIT {limit} OFFSET {offset}")

for user in PaginatedQuery(fetch_users, page_size=50):
    send_email(user)

Common Pitfalls and Best Practices

Iterator Exhaustion

The most common mistake is forgetting that iterators are single-use:

numbers = iter([1, 2, 3])

# First loop works
for n in numbers:
    print(n)  # 1, 2, 3

# Second loop prints nothing—iterator is exhausted
for n in numbers:
    print(n)  # Nothing!

If you need multiple passes, either recreate the iterator or convert to a list first.

Iterators vs. Generators

For most custom iteration needs, generators are simpler:

# Iterator class (verbose)
class Squares:
    def __init__(self, n):
        self.n = n
        self.i = 0
    
    def __iter__(self):
        return self
    
    def __next__(self):
        if self.i >= self.n:
            raise StopIteration
        result = self.i ** 2
        self.i += 1
        return result

# Generator function (concise)
def squares(n):
    for i in range(n):
        yield i ** 2

Use iterator classes when you need complex state management or want to implement additional methods. Use generators for straightforward sequences.

Don’t Modify While Iterating

Modifying a collection while iterating over it causes undefined behavior:

# This will cause problems
numbers = [1, 2, 3, 4, 5]
for n in numbers:
    if n % 2 == 0:
        numbers.remove(n)  # Don't do this

# Instead, iterate over a copy or use a comprehension
numbers = [n for n in numbers if n % 2 != 0]

Understanding iter() and next() isn’t just academic—it’s the difference between code that works and code that scales. These primitives let you process data streams of any size, build responsive pagination systems, and write memory-efficient pipelines. Master them, and you’ll write Python that handles real-world data volumes without breaking a sweat.