Iterators & Generators

Iterables and iterators

Two words that sound alike mean different things, and keeping them apart is the key to this module.

• An iterable is anything you can loop over: lists, strings, dicts, sets, files, range objects. Technically, an object is iterable if iter(obj) works, which means its type defines __iter__ (or the older __getitem__ fallback from M11).
• An iterator is the object that does the actual stepping. It remembers a position and hands out one item each time you call next() on it, raising StopIteration when it's exhausted. Its type defines __next__, and its __iter__ returns itself.

An analogy: a book is an iterable; a bookmark is an iterator. Many readers can each have their own bookmark in the same book, and a bookmark only moves forward.

The exhaustion trap

A list can be looped over any number of times, because each loop calls iter() and gets a fresh iterator. An iterator, once consumed, stays empty. Many built-ins return iterators (map, filter, zip, enumerate, reversed, file objects, generators), so this surprises everyone at least once:

Remember: an iterable can produce iterators; an iterator produces items, once, until StopIteration.

The for loop, desugared

"Syntactic sugar" is syntax that is shorthand for something longer. The for loop is sugar for a while loop over an iterator. These two programs do exactly the same thing:

for item in iterable:
    body(item)

_it = iter(iterable)
while True:
    try:
        item = next(_it)
    except StopIteration:
        break
    body(item)

Step through the desugared version. Watch the iterator's hidden position advance and the loop end when StopIteration is caught:

This is the whole iterator protocol: __iter__ to get an iterator, __next__ to get items, StopIteration to finish. Every consumer of iterables, including list(), sum(), sorted(), in, unpacking (a, b = pair), *args, comprehensions, and str.join, speaks this same protocol. So any object you make iterable works with all of them.

Remember: for calls iter() once, then next() repeatedly until StopIteration.

Writing an iterator class

To make your own iterator, implement the protocol directly: __next__ returns the next item or raises StopIteration, and __iter__ returns self.

A container that should be loopable many times must not be its own iterator. Instead, its __iter__ returns a new iterator each time. This is the iterable/iterator split from the first section, in code:

That is a lot of code for "go through a list". Generators, next, collapse it to three lines.

Remember: iterators implement __next__ and return self from __iter__; reusable containers return a fresh iterator from __iter__.

Generator functions and yield

A generator function is any function whose body contains yield. Calling it does not run the body. Instead it returns a generator: an iterator that runs the body lazily. Each next() runs the code until the next yield, hands out the yielded value, and then pauses, freezing the function's local variables and position. The next next() resumes right after the yield. When the function returns, the generator raises StopIteration.

Step through the trace and watch the generator's frame appear, pause with its locals intact, and resume, interleaved with the caller's code:

Notice that calling countdown(3) executed no line of the body; the first line ran only on next(gen). That's lazy evaluation: work is done only when a value is requested. The return value isn't yielded; it rides along inside the StopIteration exception (used by yield from, below).

Now the Playlist from the previous section, with a generator as its __iter__. Each call creates a new generator, so it's automatically reusable:

Remember: calling a generator function returns a paused generator; each next() runs to the next yield; locals survive between steps.

Generator expressions and memory

A generator expression is a comprehension in parentheses: (n * n for n in nums). It produces a generator instead of a list. When it's the only argument to a function, the extra parentheses can be dropped: sum(n * n for n in nums).

The difference is memory. A list comprehension builds every element at once; a generator expression stores only its current state, regardless of how many items it will produce.

sys.getsizeof reports only the container itself; the million int objects the list points to take tens of megabytes more. The generator's size is the same for ten items or ten billion.

any stops at the first true value, and because the generator is lazy, check(5) and check(7) never run. With a list comprehension inside any([...]), all four checks would run first.

Remember: parentheses make it lazy and constant-memory; use generator expressions for single-pass consumers.

yield from, send, and close

Delegating with yield from

yield from iterable yields every item of another iterable, as if you'd written for x in iterable: yield x. With a sub-generator it does more: it passes send() and close() through, and its value is the sub-generator's return value. It's the natural tool for recursive generators, such as walking a tree:

send() and close() (briefly)

A yield is also an expression: received = yield value. Calling gen.send(x) resumes the generator and makes that yield evaluate to x. (next(gen) is the same as gen.send(None).) You must start a generator with next() before you can send to it. gen.close() raises GeneratorExit at the paused yield so finally blocks run and resources are released.

Remember: yield from delegates to another iterable (and returns a sub-generator's result); send passes values in; close triggers cleanup.

Infinite generators and lazy pipelines

Because generators produce values on demand, they can describe sequences with no end. The consumer decides when to stop, usually with itertools.islice, a break, or a function like next or any.

Pipelines over big files

The real professional payoff is processing data that doesn't fit in memory. A file object is already a lazy iterator over lines. Chain generator stages, each taking an iterator and yielding transformed items, and the whole pipeline holds just one line at a time, whether the file has 100 lines or 100 million.

def read_lines(path):
    with open(path, encoding="utf-8") as f:   # the file closes when the generator finishes
        for line in f:                          # one line in memory at a time
            yield line.rstrip("\n")

def parse(lines):
    for line in lines:
        level, _, message = line.partition(" ")
        yield {"level": level, "message": message}

def errors_only(records):
    return (r for r in records if r["level"] == "ERROR")

# Nothing is read until the final consumer pulls values through:
pipeline = errors_only(parse(read_lines("app.log")))
print(sum(1 for _ in pipeline))

The visualizer below shows the same idea in a data pipeline. Switch between batch mode (each stage processes everything before the next starts, like chaining lists) and streaming mode (records flow through one at a time, like chaining generators). Notice how much sooner the first record arrives when streaming.

Try a runnable version, with an in-memory "file" (an io.StringIO behaves like an open text file) and a print in each stage so you can see the interleaving:

Remember: infinite generators are fine because consumers stop them; chain generator stages for constant-memory processing of any amount of data.

The itertools toolkit

The standard library's itertools module is a set of fast, memory-efficient building blocks for iterators, implemented in C. Learning them saves you from writing (and debugging) your own loops.

Remember: reach for itertools before writing a loop; groupby needs sorted input.

Strictness: zip, batched, and map

Lazy tools have one dangerous default: when input lengths don't line up, they quietly stop or produce a short final chunk. zip(names, scores) silently drops extra names if scores is shorter, hiding a data bug. Modern Python adds strict=True options that turn the mismatch into a ValueError:

• zip(a, b, strict=True) (3.10+): raise if the iterables have different lengths.
• itertools.batched(it, n, strict=True) (3.13+): raise if the final batch is incomplete.
• map(func, a, b, strict=True) (3.14+): raise if the iterables have different lengths.

Remember: pass strict=True to zip, batched (3.13+), and map (3.14+) when lengths must match.

Practice

class Steps:
    def __init__(self, start, stop, step):
        if step <= 0:
            raise ValueError("step must be positive")
        self.start = start
        self.stop = stop
        self.step = step
        self._i = 0

    def __iter__(self):
        return self

    def __next__(self):
        value = self.start + self._i * self.step
        if value >= self.stop:
            raise StopIteration
        self._i += 1
        return value

from itertools import islice


def collatz(n):
    while True:
        yield n
        n = n // 2 if n % 2 == 0 else 3 * n + 1


def take(n, iterable):
    return list(islice(iterable, n))


def collatz_length(n):
    for count, value in enumerate(collatz(n), start=1):
        if value == 1:
            return count

from collections import deque


def windows(iterable, size):
    window = deque(maxlen=size)
    for item in iterable:
        window.append(item)
        if len(window) == size:
            yield tuple(window)


def moving_average(values, size):
    for w in windows(values, size):
        yield sum(w) / size

def flatten(obj):
    if isinstance(obj, (list, tuple, set)):
        for item in obj:
            yield from flatten(item)
    else:
        yield obj

LEVELS = ["DEBUG", "INFO", "WARN", "ERROR"]


def parse(lines):
    for line in lines:
        line = line.strip()
        if not line or line.startswith("#"):
            continue
        timestamp, level, message = line.split(" ", 2)
        yield timestamp, level, message


def at_least(records, level):
    threshold = LEVELS.index(level)
    for record in records:
        if LEVELS.index(record[1]) >= threshold:
            yield record


def messages(records):
    for _, _, message in records:
        yield message


def count_levels(records):
    counts = {}
    for _, level, _ in records:
        counts[level] = counts.get(level, 0) + 1
    return counts

from itertools import groupby, chain, repeat


def rle_encode(text):
    return [(ch, sum(1 for _ in group)) for ch, group in groupby(text)]


def rle_decode(pairs):
    return "".join(chain.from_iterable(repeat(ch, n) for ch, n in pairs))

Mini-project: a log-processing pipeline

pixi init logpipe
cd logpipe
mkdir -p src
pixi run python src/make_logs.py 1000000 > big.log      # step 1: a generator that writes fake logs
pixi run python src/logpipe.py big.log --level WARN --top 5

Quiz

Recap

• An iterable gives iterators (__iter__); an iterator gives items (__next__) and raises StopIteration.
• for x in it = iter() once, then next() until StopIteration.
• Iterators are single-use; containers should return a fresh iterator from __iter__.
• Generator functions (yield) pause and resume, keeping their locals; calling one runs nothing yet.
• Generator expressions are lazy and constant-memory, ideal for one-pass consumers.
• yield from delegates; send passes values in; close runs cleanup.
• Infinite generators are safe; consumers decide when to stop. Chain stages for big-data pipelines.
• itertools provides count, cycle, islice, chain, groupby (sorted input!), accumulate, pairwise, batched, product, permutations, combinations.
• Use strict=True with zip, batched (3.13+), and map (3.14+).

Going further

• Iterator types and the glossary on generators
• Functional Programming HOWTO: iterators and generators
• itertools, including the "Itertools Recipes" section
• PEP 255 (simple generators), PEP 380 (yield from)
• David Beazley: Generator Tricks for Systems Programmers
• Next: M15 Decorators & Context Managers