Decorators & Context Managers

Functions are objects

In Python, def is an assignment statement in disguise. It builds a function object and binds it to a name, exactly like x = 42 binds an int. Because a function is an ordinary object you can do everything with it that you can do with any other object: put it in a list, store it in a dict, pass it as an argument, return it from another function, and read its attributes. Languages with this property are said to have first-class functions.

A function that takes or returns another function is called a higher-order function. You have used several already: sorted(words, key=len) passes the function len; map(str.upper, words) passes a method. Decorators are simply higher-order functions with a special syntax.

Functions can also be created inside other functions and returned. Each call to the outer function produces a brand-new function object:

Remember: a function name is just a reference to a function object. fn refers; fn() calls.

Closures and cell variables

Look again at make_greeter. By the time we call hello("Ada"), make_greeter has returned and its local variable greeting should be gone. Yet greet still knows it was "Hello". How?

When the compiler sees that an inner function uses a variable from an enclosing function, it stores that variable in a cell: a tiny box that lives on the heap instead of in the outer function's stack frame. The inner function keeps a reference to the cell. A function bundled with the cells it needs is a closure. The variable is called a free variable of the inner function and a cell variable of the outer one.

Step through this program. Watch count survive after make_counter returns, and notice that the two counters have independent cells.

nonlocal

Reading a variable from an enclosing scope needs nothing special (that is the "E" in the LEGB rule from M05). Assigning to it is different: count += 1 is an assignment, and an assignment anywhere in a function makes the name local to that function. Without nonlocal count, Python would treat count as a new local and fail with UnboundLocalError when it tries to read it before assigning. nonlocal says "this name refers to the variable in the nearest enclosing function".

The late-binding trap

A closure captures the variable, not its value at the moment the inner function was created. The value is looked up when the inner function runs. This surprises almost everyone once:

callbacks = [lambda i=i: i for i in range(3)]    # [0, 1, 2]

Remember: a closure is a function plus the cells of the enclosing variables it uses. Use nonlocal to rebind them. Closures see the variable's current value, not a snapshot.

Decorators, desugared

A decorator is a callable that takes a function and returns a replacement for it. The @ syntax is pure shorthand. These two snippets are exactly equivalent:

@announce
def greet(name):
    return f"Hi {name}"

def greet(name):
    return f"Hi {name}"
greet = announce(greet)

The replacement is almost always a closure, conventionally called wrapper, that does something before and/or after calling the original. It accepts *args, **kwargs so it can wrap a function of any signature and forwards them unchanged:

def announce(func):                      # runs ONCE, when the decorated def executes
    def wrapper(*args, **kwargs):        # runs on EVERY call
        print(f"calling {func.__name__}")
        result = func(*args, **kwargs)   # call the original
        print(f"{func.__name__} returned {result!r}")
        return result                    # don't forget to return it!
    return wrapper

A crucial detail: the decorator itself runs at definition time, when Python executes the def statement (usually at import). The wrapper runs at call time. Predict the order of the printed lines:

Remember: @deco above def f means f = deco(f). Decorator body: once, at definition. Wrapper body: every call.

Preserving identity with functools.wraps

Our wrapper has a problem: after decoration, greet.__name__ is "wrapper" and its docstring is gone. That breaks help(), makes tracebacks and logs confusing, and confuses tools such as pytest and documentation generators that rely on a function's metadata.

functools.wraps(func) is itself a decorator (with an argument!) that you apply to the wrapper. It copies __name__, __qualname__, __doc__, __module__, __annotations__, and __dict__ from the original, and sets wrapper.__wrapped__ = func so tools such as inspect.signature can find the real signature.

Remember: always decorate the wrapper with @functools.wraps(func); the original is then available as __wrapped__.

Decorators with arguments: three layers

How would you write @repeat(3)? Read the desugaring carefully: @repeat(3) means greet = repeat(3)(greet). So repeat(3) is called first and must return a decorator. That gives three nested functions, each with one job:

1. The factory repeat(times) receives the configuration and returns a decorator.
2. The decorator decorate(func) receives the function and returns a wrapper.
3. The wrapper wrapper(*args, **kwargs) runs on every call. It closes over both times and func.

Optional arguments: @timed and @timed(unit="ms")

Sometimes you want a decorator to work both bare and with arguments. The idiom: make the function the first parameter with a default of None, and make every option keyword-only. If Python passed a function, you were used bare; otherwise you were called with options and must return a decorator.

def timed(func=None, *, unit="s"):
    def decorate(f):
        @functools.wraps(f)
        def wrapper(*args, **kwargs):
            start = time.perf_counter()
            try:
                return f(*args, **kwargs)
            finally:
                elapsed = time.perf_counter() - start
                wrapper.last_elapsed = elapsed * (1000 if unit == "ms" else 1)
        return wrapper
    return decorate(func) if func is not None else decorate

Remember: @factory(args) means f = factory(args)(f): three layers, factory → decorator → wrapper.

Stacking, class decorators, and classes as decorators

Stacking order

You can stack several decorators. They are applied bottom-up (the one nearest the def wraps first) and therefore run top-down on each call, like layers of an onion:

@bold
@italic
def hi(): ...
# means: hi = bold(italic(hi))

Order matters in practice. If you stack @cache above @authorize, a cached result can be returned to a user who would have failed the authorization check, because the cache answers before the check runs. Put checks outside (above) caches.

Class decorators

A decorator can also be applied to a class: @deco class C: ... means C = deco(C). You have used one already: @dataclass inspects the class's annotations and adds __init__, __repr__, and __eq__. Class decorators usually modify the class and return the same object, or register it somewhere.

Classes as decorators

Anything callable can be a decorator, including an instance of a class that defines __call__. This is handy when the decorator needs visible state.

Remember: stacked decorators apply bottom-up and run top-down. Classes can be decorated, and callable objects can be decorators.

Real-world decorators

Decorators shine for cross-cutting concerns: behavior that many unrelated functions need but that is not part of what any of them is about. Here are the five you will meet most often.

def timed(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        start = time.perf_counter()        # monotonic, high resolution
        try:
            return func(*args, **kwargs)
        finally:                           # record even if func raises
            wrapper.last_seconds = time.perf_counter() - start
    wrapper.last_seconds = 0.0
    return wrapper

def retry(times=3, exceptions=(Exception,), delay=0.1, sleep=time.sleep):
    def decorate(func):
        @functools.wraps(func)
        def wrapper(*args, **kwargs):
            for attempt in range(1, times + 1):
                try:
                    return func(*args, **kwargs)
                except exceptions:
                    if attempt == times:
                        raise                       # give up: re-raise the last error
                    sleep(delay * 2 ** (attempt - 1))  # exponential backoff
        return wrapper
    return decorate

def memoize(func):
    cache = {}
    @functools.wraps(func)
    def wrapper(*args):
        if args not in cache:
            cache[args] = func(*args)
        return cache[args]
    return wrapper

# In real code, use the standard library:
@functools.cache          # unbounded; @functools.lru_cache(maxsize=128) is bounded
def fib(n): return n if n < 2 else fib(n - 1) + fib(n - 2)

COMMANDS = {}

def command(name):
    def decorate(func):
        COMMANDS[name] = func
        return func          # return the original unchanged
    return decorate

@command("greet")
def greet(who): return f"hello {who}"

COMMANDS["greet"]("Ada")

def require_role(role):
    def decorate(func):
        @functools.wraps(func)
        def wrapper(user, *args, **kwargs):
            if role not in user.roles:
                raise PermissionError(f"{user.name} needs role {role!r}")
            return func(user, *args, **kwargs)
        return wrapper
    return decorate

@require_role("admin")
def delete_account(user, account_id): ...

Run them together. The retry demo uses a fake sleep that just records delays, so it runs instantly and deterministically:

Remember: a decorator is the right tool when the same before/after behavior would otherwise be copy-pasted into many functions.

Context managers: guaranteed cleanup

Many resources must be released no matter what: files closed, locks released, database transactions committed or rolled back, temporary settings restored. You met with open(...) as f: in M09. A context manager is any object that implements two methods, and the with statement guarantees the second runs even when the block raises.

with manager as value:
    body

# is roughly equivalent to:
value = manager.__enter__()
try:
    body
except BaseException as exc:
    if not manager.__exit__(type(exc), exc, exc.__traceback__):
        raise                         # __exit__ returned falsy: propagate
else:
    manager.__exit__(None, None, None)

• __enter__(self) does the setup. Whatever it returns is bound to the name after as (often self, sometimes another object such as the file).
• __exit__(self, exc_type, exc_value, traceback) does the cleanup. If the block finished normally all three are None. If it raised, they describe the exception.
• If __exit__ returns a truthy value, the exception is suppressed and execution continues after the with block. Return None (the default) or False to let it propagate. Suppressing should be rare and deliberate.

Remember: __enter__ sets up and returns the as value; __exit__ always runs and suppresses the exception only if it returns something truthy.

contextlib: context managers the easy way

@contextmanager

Writing a class for every context manager is verbose. contextlib.contextmanager turns a generator function (M14) into a context manager: code before yield is the setup, the yielded value is the as value, and code after yield is the cleanup. If the block raises, the exception is re-raised at the yield, so wrap it in try/finally to make cleanup unconditional.

suppress, redirect_stdout, and friends

• contextlib.suppress(*exceptions): ignore specific exceptions. Clearer than try/except: pass for the rare cases where ignoring really is correct, such as deleting a file that may not exist.
• contextlib.redirect_stdout(file) / redirect_stderr: temporarily send print output somewhere else, such as an io.StringIO. This is how this course's exercise checker captures your output.
• contextlib.chdir(path): temporarily change the working directory.
• contextlib.nullcontext(value): a do-nothing context manager for "maybe use a lock" code paths.

ExitStack: a dynamic number of context managers

What if you need to open a list of files whose length you only know at run time? You cannot write a variable number of nested with statements. ExitStack is a context manager that collects others: stack.enter_context(cm) enters one and schedules its exit. When the stack closes, it exits them all in reverse order, even if one of the later setups fails. stack.callback(fn, *args) schedules any cleanup function.

Remember: prefer @contextmanager with try/finally around a single yield; reach for ExitStack when the number of resources is dynamic.

Preview: async context managers

Asynchronous code (Part 7) needs setup and cleanup that can themselves wait, for example opening a network connection. The protocol is the same idea with coroutine methods __aenter__ and __aexit__, used with async with. contextlib.asynccontextmanager is the async twin of @contextmanager.

Practice

def make_counter(start=0, step=1):
    count = start

    def counter():
        nonlocal count
        count += step
        return count

    return counter

import functools

def log_calls(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        result = func(*args, **kwargs)
        wrapper.calls.append((func.__name__, args, kwargs, result))
        return result

    wrapper.calls = []
    return wrapper

@log_calls
def add(a, b=0):
    """Add two numbers."""
    return a + b

import functools

def retry(times, exceptions=(Exception,), sleep=None, delay=1.0):
    def decorate(func):
        @functools.wraps(func)
        def wrapper(*args, **kwargs):
            for attempt in range(1, times + 1):
                try:
                    return func(*args, **kwargs)
                except exceptions:
                    if attempt == times:
                        raise
                    if sleep is not None:
                        sleep(delay * 2 ** (attempt - 1))
        return wrapper
    return decorate

COMMANDS = {}

def command(name):
    def decorate(func):
        if name in COMMANDS:
            raise ValueError(f"command {name!r} already registered")
        COMMANDS[name] = func
        return func
    return decorate

def dispatch(line):
    word, *args = line.split()
    func = COMMANDS.get(word)
    if func is None:
        return f"unknown command: {word}"
    return func(*args)

@command("add")
def add(a, b):
    return int(a) + int(b)

@command("echo")
def echo(*words):
    return " ".join(words)

from contextlib import contextmanager

class Ignoring:
    def __init__(self, *exc_types):
        self.exc_types = exc_types
        self.caught = None

    def __enter__(self):
        return self

    def __exit__(self, exc_type, exc, tb):
        if exc_type is not None and issubclass(exc_type, self.exc_types):
            self.caught = exc
            return True
        return False

@contextmanager
def temporary_attr(obj, name, value):
    old = getattr(obj, name)
    setattr(obj, name, value)
    try:
        yield obj
    finally:
        setattr(obj, name, old)

Mini-project: a resilience toolkit

pixi init resilience && cd resilience
mkdir -p src/resilience tests
pixi add pytest
# then add your package as an editable install in pixi.toml (see M08):
#   [pypi-dependencies]
#   resilience = { path = ".", editable = true }
pixi run pytest

Quiz

Recap

• Functions are first-class objects: fn refers, fn() calls.
• A closure is a function plus the cells of the enclosing variables it uses; nonlocal rebinds them; closures see the current value (the late-binding trap).
• @deco means f = deco(f); the decorator runs at definition time, the wrapper on each call.
• Always use @functools.wraps(func). Decorators with arguments need three layers.
• Stacked decorators apply bottom-up and run top-down; order matters for checks and caches.
• Context managers pair setup with guaranteed cleanup. __exit__ suppresses only by returning truthy.
• @contextmanager + try/finally around one yield; ExitStack for dynamic sets; suppress and redirect_stdout for common jobs; async with for async setup.

Going further

• Glossary: decorator and function definitions in the language reference
• functools.wraps and contextlib
• Data model: with statement context managers
• PEP 318 (decorators) and PEP 343 (the with statement)
• Next: M16 Type Hints & Static Typing, where you will learn to type decorators with ParamSpec.