Standard Library Power Tools

collections: specialized containers

The collections module provides containers that encode common patterns. Each replaces a few lines of fiddly code with one well-tested, usually C-implemented type.

itertools: lazy building blocks

itertools provides fast, memory-efficient iterator building blocks (M14 and M18 introduced the idea). Everything is lazy: nothing is computed until you iterate. The docs' "recipes" section shows how to combine them; many recipes are now built-ins or available in the third-party more-itertools package (pixi add more-itertools).

Dates, times, and time zones

Time handling is where confident programmers write subtle bugs. The foundation is the difference between two kinds of datetime:

The professional rule: store and compute in UTC, convert to local time only for display (and keep the user's IANA zone name, like "America/New_York", not just an offset like -05:00, since offsets change with DST and with legislation).

zoneinfo.ZoneInfo (3.9+) uses the IANA time-zone database: the operating system's copy, or the tzdata package where the OS has none (Windows, minimal containers, and this browser).

DST pitfalls: gaps, folds, and wall-clock arithmetic

Daylight saving time creates two kinds of trouble:

• Gaps: when clocks spring forward, some local times don't exist (in New York on 2026-03-08, 02:30 never happens).
• Folds: when clocks fall back, some local times happen twice (01:30 on 2026-11-01 in New York). The fold attribute (0 = first occurrence, 1 = second) disambiguates.
• Arithmetic is wall-clock arithmetic: adding timedelta(days=1) to an aware datetime in a ZoneInfo zone keeps the same wall time and zone, which may be 23 or 25 real hours later. Subtracting two datetimes in the same zone object also ignores DST. For elapsed real time, convert to UTC first.

Regular expressions in depth

A regular expression is a small pattern language for matching text. Python's re module compiles a pattern to a state machine and runs it with backtracking. Always write patterns as raw strings (r"\d+") so backslashes reach re unchanged.

Functions: re.match (anchored at the start), re.fullmatch (whole string), re.search (anywhere), re.findall (list of strings or group tuples), re.finditer (match objects, lazily), re.sub (replace; the replacement can be a function), re.split. Flags: re.IGNORECASE, re.MULTILINE, re.DOTALL, re.VERBOSE, re.ASCII. re.compile builds a reusable pattern object; the module also caches recently used patterns, but compiling makes intent and reuse explicit.

Readable patterns with re.VERBOSE

With re.VERBOSE (re.X), whitespace in the pattern is ignored and # starts a comment, so a complex pattern can be laid out and documented like code. (Escape spaces you mean literally as \ or use [ ].)

argparse: command-line interfaces

argparse turns sys.argv into a validated namespace and generates --help and error messages for free. The essentials: positional arguments, options with type, default, choices, nargs, and action ("store_true", "count", "append"); mutually exclusive groups; and subcommands with add_subparsers, which is how tools like git and pixi are structured.

sqlite3: a real SQL database in the standard library

SQLite is an embedded SQL database: a single file (or ":memory:"), no server, transactional, and astonishingly robust. It's ideal for local tools, caches, tests, and prototypes (M44 goes deeper). Three habits matter most:

1. Always use placeholders (? or :name), never f-strings, to pass values. Formatting user input into SQL is how SQL injection happens.
2. Use the connection as a context manager for transactions: with conn: commits on success and rolls back on an exception. (It does not close the connection; use contextlib.closing or call close().)
3. Set row_factory = sqlite3.Row to access columns by name.

IDs, randomness, and hashing

uuid: uuid4 vs 3.14's uuid7

A UUID is a 128-bit identifier you can generate anywhere without coordination. The two you should know:

• uuid4(): 122 random bits. Unpredictable and collision-safe, but random order: as database primary keys they scatter inserts across the index, which hurts B-tree locality.
• uuid7() 3.14: a 48-bit Unix timestamp in milliseconds followed by random bits (RFC 9562). IDs generated later sort later, so they index like auto-increment integers while remaining globally unique. They do reveal creation time, so don't use them where that's sensitive.

Avoid uuid1() (embeds your MAC address). uuid5(namespace, name) gives a deterministic ID for a name. 3.14 also adds uuid6, uuid8, and the uuid.NIL/uuid.MAX constants.

secrets vs random

random uses the Mersenne Twister: fast, reproducible with a seed, and predictable (observe 624 outputs and you can predict the rest). Perfect for simulations, sampling, shuffling test data, and games. For anything an attacker must not guess (passwords, session tokens, reset links, API keys), use secrets, which draws from the operating system's cryptographically secure generator.

hashlib and hmac

Files: shutil, tempfile, and glob

pathlib (M09) handles paths; these modules handle the rest:

• shutil: copy2 (with metadata), copytree, move, rmtree (careful!), make_archive/unpack_archive (3.14 adds zstdtar), which (find an executable on PATH), disk_usage. 3.14 also adds Path.copy() and Path.move().
• tempfile: TemporaryDirectory() and NamedTemporaryFile() create secure, uniquely named temporary locations and delete them when the with block exits. Never build temp paths by hand (/tmp/myapp.txt): it races with other processes and is a security hole.
• glob.glob("**/*.py", recursive=True) or Path.rglob("*.py"): find files by pattern.

enum, textwrap, and string

Enums (M13) give names to a fixed set of values. Recap of the kinds: Enum (opaque members), IntEnum and StrEnum (members are ints/strs, handy for JSON and CLI choices), Flag (combinable bits with |), auto() for values, and @unique to forbid aliases. Iterating an enum class, and Color["RED"], work thanks to its metaclass (M36).

textwrap formats prose: dedent (strip common indentation from triple-quoted strings), fill/wrap, shorten (truncate at a word boundary with a placeholder), and indent. string offers constants (ascii_letters, digits, punctuation) and string.Template ($name substitution, safe for user-supplied templates because it can't evaluate expressions).

logging, and why Part 6 uses structlog

Beginner code uses print(); production code uses logging, because logs need levels (debug noise off in production), timestamps, the source module, routing (console, file, log aggregator), and configuration without code changes. The standard logging module provides all of that: loggers named by module (logging.getLogger(__name__)), levels, handlers, and formatters.

The weakness of classic logging is that messages are text: to answer "how many 500s per path in the last hour?", a log aggregator must parse human sentences. Structured logging emits events as key-value data (usually JSON lines): {"event": "request", "path": "/api", "status": 500, "ms": 31}. structlog makes that ergonomic and is the course's standard from Part 6 on (M42):

Recap: t-strings and string.templatelib

Practice

import re
from collections import Counter

def top_words(text, n, stopwords=()):
    stop = set(stopwords)
    counts = Counter(w for w in re.findall(r"[a-z']+", text.lower()) if w not in stop)
    return sorted(counts.items(), key=lambda kv: (-kv[1], kv[0]))[:n]

from collections import deque
from itertools import count, islice

def windows(iterable, n):
    window = deque(maxlen=n)
    for item in iterable:
        window.append(item)
        if len(window) == n:
            yield tuple(window)

def sessions(times, gap):
    current = []
    for t in times:
        if current and t - current[-1] > gap:
            yield current
            current = []
        current.append(t)
    if current:
        yield current

import re

LINE = re.compile(r"""
    ^(?P<ts>\d{4}-\d\d-\d\dT\d\d:\d\d:\d\d(?:Z|[+-]\d\d:\d\d))
    \s+(?P<level>DEBUG|INFO|WARN|ERROR)
    \s+\[(?P<component>[\w.-]+)\]
    \s*(?P<tail>.*)$
""", re.VERBOSE)

FIELD = re.compile(r"(\w+)=(\S+)")

def parse_line(line):
    m = LINE.match(line)
    if m is None:
        return None
    tail = m["tail"]
    return {
        "ts": m["ts"],
        "level": m["level"],
        "component": m["component"],
        "message": FIELD.sub("", tail).strip(),
        "fields": dict(FIELD.findall(tail)),
    }

try:
    import tzdata
except ImportError:
    pass
from datetime import UTC, datetime, timedelta
from zoneinfo import ZoneInfo

def to_zone(iso_string, zone):
    dt = datetime.fromisoformat(iso_string)
    if dt.tzinfo is None:
        raise ValueError("timestamp must include an offset")
    return dt.astimezone(ZoneInfo(zone))

def real_hours(start, end):
    return (end.astimezone(UTC) - start.astimezone(UTC)) / timedelta(hours=1)

import sqlite3
from contextlib import closing

def top_customers(orders, k, min_amount=0):
    with closing(sqlite3.connect(":memory:")) as conn:
        conn.execute("CREATE TABLE orders (customer TEXT, amount REAL)")
        with conn:
            conn.executemany("INSERT INTO orders VALUES (?, ?)", orders)
        rows = conn.execute(
            "SELECT customer, SUM(amount) AS total FROM orders WHERE amount >= ? "
            "GROUP BY customer ORDER BY total DESC, customer LIMIT ?",
            (min_amount, k),
        ).fetchall()
    return [(name, total) for name, total in rows]

Mini-project: a log-parsing CLI

pixi init logtool
cd logtool
pixi add python=3.14 pytest
pixi run python -m logtool stats sample.log --tz America/New_York --top 5
pixi run python -m logtool grep sample.log --level ERROR --since 2026-07-01T08:00 --tz Europe/Oslo --format json
pixi run python -m logtool hist sample.log --bucket 15m --tz Asia/Kolkata
pixi run python -m pytest -q

Quiz

Recap

• deque, Counter, defaultdict, ChainMap, and namedtuple replace common hand-written patterns.
• itertools is lazy: pairwise, batched, accumulate, chain, and groupby (consecutive keys) cover most needs.
• Use aware datetimes, store UTC, display local; ZoneInfo arithmetic is wall-clock, so convert to UTC for elapsed time; mind gaps and folds.
• Compile regexes, use named groups and re.VERBOSE, use lookarounds for conditions, and beware catastrophic backtracking.
• argparse subcommands with main(argv) -> int; 3.14 adds color and suggest_on_error.
• sqlite3: placeholders always, with conn: for transactions, sqlite3.Row for names.
• uuid7 for sortable IDs, secrets for tokens, HMAC for message authentication, slow KDFs for passwords.
• tempfile, shutil, glob, enum, textwrap, logging, and t-strings round out the toolkit.

Going further

• collections and itertools recipes
• zoneinfo and aware and naive objects
• Regular Expression HOWTO
• argparse, sqlite3, uuid, secrets
• string.templatelib
• Next: Project 5: Async Web Crawler