Part 2 · Intermediate Python · Module 17

Testing with pytest

Automated tests are how professionals change code without fear. This module teaches pytest, the standard Python test framework, from your first assert to fixtures, parametrization, mocking, coverage gates, and test-driven development.

⏱ ~3 hoursPrerequisites: M07 Errors & Exceptions, M15 Decorators & Context Managers

Why test?

A test is code that runs other code and checks the result automatically. You have been testing all along: every time you ran a script and eyeballed the output, that was a manual test. Automated tests make that check repeatable, fast, and exact.

The testing pyramid

E2E Integration Unit tests slow, brittle, realistic fast, focused, many
Most tests should be fast unit tests; fewer integration tests; a handful of end-to-end tests.

Tests are just functions with asserts

In pytest, a test is a function whose name starts with test_ and that uses plain assert statements. If the function returns normally, the test passes. If an assert fails (raising AssertionError) or any other exception escapes, it fails.

A good test has three parts, often called Arrange, Act, Assert (AAA): set up the inputs, call the code under test, check the outcome. Keep each test focused on one behavior and name it after that behavior.

Step through one test and watch the three phases happen. When the assert holds, the function simply returns: that is a pass.

To demystify pytest, here is a 20-line test runner that does the core of what it does: discover functions named test_*, run each one, and report passes and failures with the assertion message. Run it, then break average and run it again.

Remember: a test is a test_ function that arranges, acts, and asserts. Failing means an exception escaped.

pytest: setup, discovery, and assertion rewriting

Installing and running with Pixi

pytest is a development dependency. Add it (and the coverage plugin you will use later) to your workspace. Put them in a feature if you keep separate production and development environments.

pixi add pytest pytest-cov                  # or: pixi add --feature test pytest pytest-cov
pixi run pytest                             # run every test
pixi run pytest -v                          # verbose: one line per test
pixi run pytest tests/unit/test_stats.py    # one file
pixi run pytest -k "average and not empty"  # tests whose names match an expression
pixi run pytest -x --lf                     # stop at the first failure; rerun only last failures
[tasks]
test = "pytest tests/unit"
test-integration = "pytest tests/integration"
cov = "pytest tests --cov=src --cov-report=term-missing --cov-fail-under=90"

Discovery rules

pytest collects tests automatically. By default it searches the directories you give it (or testpaths from configuration) for:

[tool.pytest.ini_options]
testpaths = ["tests"]
addopts = "-v --tb=short"
markers = ["integration: tests that touch real files, databases, or networks"]

Assertion rewriting

With plain Python, a failing assert a == b just says AssertionError. pytest rewrites the assert statements in test modules when it imports them, so a failure shows the values of every sub-expression. You get this for free, which is why pytest tests use bare assert instead of self.assertEqual-style methods:

def average(xs):
    return sum(xs) / len(xs)


def test_average_of_three():
    assert average([1, 2, 3]) == 2


def test_average_rounding():
    scores = {"ada": [90, 85], "alan": [70, 81]}
    assert average(scores["alan"]) == 75
$ pixi run pytest tests/test_demo.py
collected 2 items

tests/test_demo.py .F                                                    [100%]

=================================== FAILURES ===================================
____________________________ test_average_rounding _____________________________

    def test_average_rounding():
        scores = {"ada": [90, 85], "alan": [70, 81]}
>       assert average(scores["alan"]) == 75
E       assert 75.5 == 75
E        +  where 75.5 = average([70, 81])

tests/test_demo.py:11: AssertionError
=========================== short test summary info ============================
FAILED tests/test_demo.py::test_average_rounding - assert 75.5 == 75
========================= 1 failed, 1 passed in 0.08s ==========================

Read failure output from the bottom up: the summary names the test, the E lines show the actual values, and the > marks the failing line.

Fixtures: reusable setup and teardown

Tests often need the same setup: a sample object, a temporary directory, a database connection. A fixture is a function decorated with @pytest.fixture that provides such a value. A test asks for a fixture simply by having a parameter with the fixture's name; pytest calls the fixture and passes in the result. This is dependency injection by name.

import pytest
from shop.cart import Cart


@pytest.fixture
def cart() -> Cart:
    c = Cart()
    c.add("apple", price=0.5, qty=4)
    return c


def test_total(cart: Cart) -> None:       # pytest sees "cart" and injects the fixture
    assert cart.total() == 2.0


def test_remove(cart: Cart) -> None:      # a fresh Cart: fixtures run again per test by default
    cart.remove("apple")
    assert cart.total() == 0

Teardown with yield and scopes

If a fixture yields instead of returning, the code after yield runs after the test finishes, pass or fail. It is a context manager in disguise (compare @contextmanager in M15). The scope argument controls how often the fixture is created: "function" (default, once per test), "class", "module", "package", or "session" (once per run). Use wide scopes only for expensive, read-only resources, since shared mutable state makes tests depend on each other.

@pytest.fixture(scope="module")
def db():
    conn = sqlite3.connect(":memory:")
    conn.execute("create table users (name text)")
    yield conn                      # tests run here
    conn.close()                    # teardown, once after the last test in the module

Fixtures can use other fixtures, and fixtures defined in a conftest.py file are available to every test in that directory and below, without importing.

Built-in fixtures you will use constantly

A fresh pathlib.Path to an empty temporary directory, unique to each test. Never write test files into your project directory.

def test_save_and_load(tmp_path):
    path = tmp_path / "notes.json"
    save({"a": 1}, path)
    assert load(path) == {"a": 1}

Temporarily replace attributes, dict items, or environment variables; everything is undone after the test.

def test_reads_api_key(monkeypatch):
    monkeypatch.setenv("API_KEY", "test-key")
    monkeypatch.setattr("shop.pricing.TAX_RATE", 0.0)
    assert load_settings().api_key == "test-key"

Captures what the code under test printed to stdout and stderr.

def test_greeting(capsys):
    main(["--name", "Ada"])
    out, err = capsys.readouterr()
    assert out == "Hello, Ada!\n"
    assert err == ""

Remember: ask for a fixture by naming it as a parameter; yield for teardown; default to function scope; reach for tmp_path, monkeypatch, and capsys.

Parametrize and marks

Copy-pasting a test to try five inputs is a smell. @pytest.mark.parametrize runs one test function once per row of arguments, and reports each row as a separate test:

import pytest

@pytest.mark.parametrize(
    ("year", "expected"),
    [
        (2024, True),     # divisible by 4
        (2023, False),
        (1900, False),    # century, not divisible by 400
        (2000, True),     # divisible by 400
    ],
    ids=["div4", "plain", "century", "div400"],
)
def test_is_leap(year: int, expected: bool) -> None:
    assert is_leap(year) is expected
$ pixi run pytest -v -k is_leap
test_dates.py::test_is_leap[div4] PASSED
test_dates.py::test_is_leap[plain] PASSED
test_dates.py::test_is_leap[century] PASSED
test_dates.py::test_is_leap[div400] PASSED

A mark is a label attached to a test with a decorator. pytest has built-in marks and lets you define your own:

The browser has no pytest, but the idea of parametrization is just a loop over cases with a helpful message per case. Predict which rows fail:

Testing exceptions

Error handling is behavior too, and it is where bugs hide because it rarely runs. Test that the right exception is raised with pytest.raises, a context manager. The optional match argument is a regular expression searched in the exception's message; as excinfo gives you the exception for further checks.

import pytest

def test_withdraw_too_much():
    account = Account(balance=10)
    with pytest.raises(InsufficientFunds, match=r"balance is 10") as excinfo:
        account.withdraw(25)
    assert excinfo.value.shortfall == 15
    assert account.balance == 10          # and the failed call changed nothing

How does pytest.raises work? It is a context manager whose __exit__ checks the exception type and suppresses it if it matches, exactly the pattern from M15. Here is a simplified version:

Mocking with unittest.mock, and when not to

A unit test should not send emails, charge credit cards, or call a weather API. A test double stands in for such a dependency. The standard library's unittest.mock provides the most flexible kind, the mock: an object that accepts any call, returns whatever you configure, and records how it was used so you can assert on it afterwards.

patch: replace a name for the duration of a test

When code looks up a dependency by name instead of receiving it as an argument, unittest.mock.patch temporarily replaces that name. The golden rule: patch where the name is looked up, not where it is defined. If shop/report.py does from datetime import date, you patch "shop.report.date", not "datetime.date".

When not to mock

Coverage, organization, and Pixi tasks

Measuring coverage

Coverage measures which lines (and, with branch coverage, which if/else paths) your tests executed. pytest-cov integrates coverage.py with pytest:

$ pixi run pytest tests --cov=src --cov-branch --cov-report=term-missing --cov-fail-under=90
Name                    Stmts   Miss Branch BrPart  Cover   Missing
-------------------------------------------------------------------
src/shop/__init__.py        2      0      0      0   100%
src/shop/cart.py           41      2     12      1    94%   57-58
src/shop/pricing.py        18      0      6      0   100%
-------------------------------------------------------------------
TOTAL                      61      2     18      1    96%
Required test coverage of 90% reached. Total coverage: 96.20%

The Missing column tells you exactly which lines never ran. --cov-fail-under=90 makes the command fail if total coverage drops below 90%, so CI can enforce it as a gate.

Organizing a test suite

src/shop/
    cart.py
    pricing.py
tests/
    conftest.py              # fixtures shared by all tests
    unit/
        conftest.py          # unit-only fixtures (fakes, sample objects)
        test_cart.py         # mirrors src/shop/cart.py
        test_pricing.py
    integration/
        conftest.py          # real resources: temp databases, files
        test_storage.py      # every test marked @pytest.mark.integration

Test-driven development: red, green, refactor

Test-driven development (TDD) flips the usual order: you write a failing test first, then the simplest code that passes it, then clean up. Repeat in tiny steps.

  1. Red: write one small test for behavior that does not exist yet. Run it and watch it fail (this proves the test can fail).
  2. Green: write the simplest code that makes it pass. Resist the urge to write more.
  3. Refactor: with all tests green, improve names, remove duplication, and simplify. Run the tests after each change.

Let us do a classic kata (a small practice problem): convert integers to Roman numerals.

def test_one():
    assert to_roman(1) == "I"
# NameError: to_roman is not defined. Red.
def to_roman(n):
    return "I"          # the simplest thing that passes. Really.
def test_three():
    assert to_roman(3) == "III"     # fails: got "I"
def to_roman(n):
    return "I" * n
def test_five_and_four():
    assert to_roman(5) == "V"
    assert to_roman(4) == "IV"

def to_roman(n):
    result = ""
    if n >= 5:
        result += "V"; n -= 5
    if n == 4:
        return result + "IV"
    return result + "I" * n
# The if-chain is a pattern: a table of values, largest first, greedily subtracted.
NUMERALS = [(1000, "M"), (900, "CM"), (500, "D"), (400, "CD"), (100, "C"), (90, "XC"),
            (50, "L"), (40, "XL"), (10, "X"), (9, "IX"), (5, "V"), (4, "IV"), (1, "I")]

def to_roman(n):
    result = []
    for value, symbol in NUMERALS:
        count, n = divmod(n, value)
        result.append(symbol * count)
    return "".join(result)

The refactor generalizes the solution, and the tests from every earlier step prove that it still works. Run the final version against the whole growing suite:

Preview: property-based testing with Hypothesis

Example-based tests check the inputs you thought of. Property-based tests state a rule that must hold for all inputs, and a library generates hundreds of inputs trying to break it. When it finds a failure, it shrinks the input to the smallest example that still fails. The Python library is Hypothesis.

Good properties: round trips (decode(encode(x)) == x), invariants (sorting keeps the length and the items), idempotence (normalize(normalize(x)) == normalize(x)), and comparison with a slow but obviously correct oracle.

Practice

Some exercises ask you to write code that passes tests; others ask you to write the tests. For the latter, the checker runs your test function against a correct implementation (it must pass) and against several buggy ones (it must catch each bug). This is called mutation testing, and it measures whether your tests actually check anything.

The tests are the specification. Write slugify(title) so that they pass: lower-case, words joined by single hyphens, only a-z and 0-9 kept, no leading or trailing hyphens. Read the hidden tests' names in the results panel after your first Check.

Hint

re.sub(r"[^a-z0-9]+", "-", title.lower()) then .strip("-").

Show solution 
import re

def slugify(title):
    return re.sub(r"[^a-z0-9]+", "-", title.lower()).strip("-")

Write check_is_leap(is_leap): a test function that receives an implementation of is_leap(year) and uses assert to check it thoroughly. It must pass for a correct implementation and raise AssertionError for each of four hidden buggy versions (one ignores the century rule, one ignores the 400 rule, one returns truthy ints instead of True/False, and one accepts years below 1 instead of raising). Think about which years distinguish each rule.

Rules: divisible by 4 → leap, except centuries (divisible by 100) are not, except multiples of 400 are. Years below 1 must raise ValueError.

Hint

You need at least: a plain leap year, a plain common year, a century (1900), and a multiple of 400 (2000), all compared with is; then a try: is_leap(0) / except ValueError / else: raise AssertionError(...) block.

Show solution 
def check_is_leap(is_leap):
    assert is_leap(2024) is True, "2024 is divisible by 4"
    assert is_leap(2023) is False, "2023 is not divisible by 4"
    assert is_leap(1900) is False, "centuries are not leap years"
    assert is_leap(2000) is True, "multiples of 400 are leap years"
    try:
        is_leap(0)
    except ValueError:
        pass
    else:
        raise AssertionError("year 0 should raise ValueError")

Write a context manager class raises(expected, match=None) that behaves like pytest.raises:

  • If the block raises expected (or a subclass), suppress it and store it in .value.
  • If match is given, re.search(match, str(exc)) must succeed, otherwise raise AssertionError.
  • If the block raises nothing, raise AssertionError with a message containing "DID NOT RAISE".
  • Any other exception type propagates unchanged.
Hint

All the logic is in __exit__(self, exc_type, exc, tb). Return False for a non-matching type, True to suppress, and raise AssertionError for the two failure cases.

Show solution 
import re

class raises:
    def __init__(self, expected, match=None):
        self.expected = expected
        self.match = match
        self.value = None

    def __enter__(self):
        return self

    def __exit__(self, exc_type, exc, tb):
        if exc_type is None:
            raise AssertionError(f"DID NOT RAISE {self.expected.__name__}")
        if not issubclass(exc_type, self.expected):
            return False
        if self.match is not None and not re.search(self.match, str(exc)):
            raise AssertionError(f"pattern {self.match!r} not found in {str(exc)!r}")
        self.value = exc
        return True

send_reminders(members, today, mailer) should email every member whose loan is due tomorrow, calling mailer.send(email, subject) once per such member with the subject "Due tomorrow", and return how many emails it sent. Members are dicts with email and due (a date).

Write check_send_reminders(send_reminders): build a Mock(spec=Mailer), call the function with a small list of members, and assert on the return value and on mailer.send's calls. It must pass for the correct version and catch three buggy ones.

Hint

Add assert count == 1 and mailer.send.assert_called_once_with("ada@example.org", "Due tomorrow"). The assert_* methods raise AssertionError on mismatch.

Show solution 
from datetime import date, timedelta
from unittest.mock import Mock, call

class Mailer:
    def send(self, email: str, subject: str) -> None: ...

def check_send_reminders(send_reminders):
    mailer = Mock(spec=Mailer)
    today = date(2026, 3, 1)
    members = [
        {"email": "ada@example.org", "due": date(2026, 3, 2)},
        {"email": "alan@example.org", "due": date(2026, 3, 5)},
    ]
    count = send_reminders(members, today, mailer)
    assert count == 1, f"expected 1 email, got {count}"
    mailer.send.assert_called_once_with("ada@example.org", "Due tomorrow")

Continue the kata. Write from_roman(s) that converts a Roman numeral string to an int. Work test by test: press Check early and make one failing test pass at a time. Invalid characters must raise ValueError. (You do not need to reject malformed but valid-character strings like "IIII".)

Hint

Walk the string left to right. If a symbol's value is less than the next symbol's value, subtract it; otherwise add it.

Show solution 
VALUES = {"I": 1, "V": 5, "X": 10, "L": 50, "C": 100, "D": 500, "M": 1000}

def from_roman(s):
    try:
        values = [VALUES[ch] for ch in s]
    except KeyError as exc:
        raise ValueError(f"invalid Roman numeral character: {exc.args[0]!r}") from None
    total = 0
    for i, value in enumerate(values):
        if i + 1 < len(values) and value < values[i + 1]:
            total -= value
        else:
            total += value
    return total

Mini-project: TDD the string calculator kata

Mini-projectString calculator, one red-green-refactor at a time

Roy Osherove's string calculator is the classic TDD kata. The rule: read only the next requirement, write a failing test for it, make it pass, refactor, commit. Do not read ahead.

pixi init calculator && cd calculator
pixi add pytest pytest-cov
mkdir -p src/calculator tests/unit
# add [pypi-dependencies] calculator = { path = ".", editable = true } and the test/cov tasks to pixi.toml
pixi run test
  1. add("") returns 0; add("4") returns 4; add("1,2") returns 3.
  2. Any number of numbers: add("1,2,3,4") returns 10.
  3. Newlines are separators too: add("1\n2,3") returns 6.
  4. Custom delimiter: add("//;\n1;2") returns 3.
  5. Negatives raise ValueError("negatives not allowed: -1, -3"), listing all of them.
  6. Numbers bigger than 1000 are ignored: add("2,1001") returns 2.
  7. Delimiters of any length: add("//[***]\n1***2***3") returns 6.
  • Every step started with a failing test that you saw fail
  • Tests use @pytest.mark.parametrize for the input/output tables
  • The negatives rule is tested with pytest.raises(ValueError, match=...)
  • A cov task enforces --cov-fail-under=90 and passes (aim for 100% with branch coverage)
  • add is refactored into small helpers (parse delimiter, split, validate) with tests still green
  • One commit per green step, with Conventional Commit messages (test:, feat:, refactor:)

Stretch: add a Hypothesis property (for any list of non-negative ints up to 1000, joining with commas and calling add equals sum); support multiple custom delimiters (//[*][%]\n1*2%3).

Quiz

Recap

Going further