Web & APIs

How HTTP works

HTTP (Hypertext Transfer Protocol) is a request/response protocol. A client (a browser, your script, another service) opens a connection to a server and sends a request; the server sends back exactly one response. Underneath, HTTP/1.1 runs over a TCP connection (usually wrapped in TLS, which makes it HTTPS); HTTP/2 and HTTP/3 change the wire format for speed but keep the same model of methods, headers, and status codes.

sequenceDiagram
  autonumber
  participant C as Client (your code)
  participant D as DNS
  participant S as API server
  C->>D: Where is api.example.com?
  D-->>C: 203.0.113.7
  C->>S: TCP + TLS handshake
  C->>S: GET /v1/notes?page=2 + headers (Accept, Authorization)
  S->>S: route, authenticate, query database
  S-->>C: 200 OK + headers (Content-Type: application/json) + JSON body
  C->>S: POST /v1/notes + JSON body
  S-->>C: 201 Created + Location: /v1/notes/42
    

An HTTP/1.1 message is plain text: a start line, header lines, a blank line, and an optional body. Here is a real request and its response:

POST /v1/notes HTTP/1.1
Host: api.example.com
Content-Type: application/json
Accept: application/json
Authorization: Bearer eyJhbGciOi...
Content-Length: 35

{"title": "Buy milk", "done": false}

HTTP/1.1 201 Created
Content-Type: application/json
Location: /v1/notes/42

{"id": 42, "title": "Buy milk", "done": false}

HTTP is stateless: each request carries everything the server needs to handle it, and the server does not remember the previous request on that connection. State that must persist between requests (who you are, what is in your cart) is sent every time, as a cookie or an Authorization header, or stored server-side in a database keyed by something the client sends. Statelessness is what lets you run ten copies of a server behind a load balancer: any copy can answer any request.

Methods and status codes

The method says what the client wants to do with the resource at the path. Two properties matter for correctness. A safe method does not change server state. An idempotent method has the same effect whether you send it once or five times, which is exactly what makes it safe to retry after a timeout.

The status code is a three-digit number; the first digit is its class:

The distinction between 4xx and 5xx drives client behavior: a 4xx means your request is wrong, so sending it again unchanged will fail again; a 5xx (and 429) may be temporary, so a retry can succeed. The standard library has every code as an enum:

Headers and JSON

Headers are metadata about the message. The ones you will meet constantly:

• Content-Type: the body's format (application/json, text/html; charset=utf-8). Accept: what the client would like back.
• Authorization: credentials, typically Bearer <token>.
• Cache-Control, ETag, If-None-Match: caching and conditional requests (a 304 response means "unchanged; reuse your copy").
• Location: where a newly created resource (201) or a redirect (3xx) lives.
• Retry-After: how long to wait after a 429 or 503. User-Agent: who is calling.

JSON (JavaScript Object Notation) is the lingua franca of APIs. It has objects, arrays, strings, numbers, booleans, and null, which map to dict, list, str, int/float, bool, and None. It has no dates, no sets, no tuples, no bytes, and object keys are always strings, so serializing real data needs choices.

URLs and urllib.parse

A URL has parts, and each part has its own encoding rules:

https://ada:secret@api.example.com:8443/v1/search?q=caf%C3%A9+au+lait&page=2#results
└─┬─┘   └───┬────┘ └──────┬──────┘ └┬─┘└───┬───┘ └──────────┬───────────────┘ └──┬──┘
scheme  userinfo        host      port   path              query              fragment

Characters with special meaning (/ ? & = # +, spaces, non-ASCII) must be percent-encoded when they appear inside a value. Building URLs with string concatenation is how a search for "R&D" silently becomes two parameters. Use urllib.parse:

urljoin follows browser rules: without a trailing slash, the last path segment is treated as a "file" and replaced. Many bugs in API clients come from exactly this.

Consuming APIs: urllib.request and httpx

The browser sandbox that runs this page blocks real network calls from Python, so the cells in this section are marked to run locally. Everything else in the module (URL handling, retries, routing, validation) runs here.

The standard library: urllib.request

urllib.request needs no installation and is fine for scripts, but it is low-level: no connection pooling across calls, awkward JSON and form handling, and no async. For real projects, use httpx.

httpx: sync and async

With httpx.AsyncClient the same calls become awaitable, so you can run many requests concurrently on one thread. This is where the event loop from M38 pays off: while one request waits for the network, the loop runs the others.

Pagination, simulated

APIs return big collections in pages. Offset pagination (?page=3&per_page=50) is simple but can skip or repeat items when data changes between requests. Cursor pagination returns an opaque token pointing at "after the last item you saw", which is stable and fast; GitHub-style APIs put the next URL in a Link header. A generator hides the paging from the caller:

Retries, backoff, and idempotency

Networks fail in transient ways: a dropped connection, a 503 during a deploy, a 429 because you are going too fast. The standard remedy is to retry with exponential backoff: wait 0.5 s, then 1 s, 2 s, 4 s..., up to a cap and a maximum number of attempts. Add jitter (a random fraction of the delay) so that a thousand clients that failed at the same moment do not all retry at the same moment too (the "thundering herd").

• Retry: connection errors, timeouts, 429, 502, 503, 504. Honor Retry-After if present.
• Do not retry: other 4xx (the request itself is wrong) and anything after a non-idempotent request might have succeeded, unless you use an idempotency key: a unique header (for example Idempotency-Key: 7c9e...) the server uses to recognize and ignore a repeated POST.

Retry logic is a perfect example of code that should take its clock as a parameter: then tests run instantly and deterministically, with no real sleeping.

Testing HTTP code without a network

Tests that hit real APIs are slow, flaky, rate-limited, and may cost money. Instead, cut the network out at a seam. The cleanest seam is dependency injection: your code receives the client (or a function that performs requests) instead of creating it, so a test can pass a fake.

With httpx you get this seam for free: every client accepts a transport, and httpx.MockTransport(handler) calls your function instead of the network. The respx library builds on that with a route-matching API and pytest fixtures:

Designing a REST API

REST (Representational State Transfer) is an architectural style for HTTP APIs. In practice it means: model your domain as resources identified by URLs (nouns, not verbs), use the HTTP methods for operations, use status codes honestly, and keep requests stateless.

• Plural nouns for collections; avoid /getNotes or /createNote.
• Version the API (/v1/ in the path or a header) so you can change it without breaking clients.
• Consistent errors: one JSON shape for all of them. RFC 9457 "Problem Details" is a standard: {"type": "...", "title": "Not found", "status": 404, "detail": "note 42 does not exist"}.
• Paginate every collection from day one; adding it later breaks clients.

Under every web framework is the same small idea: match the method and path against a table of routes, extract path parameters, validate the body, call a handler, and turn its result (or its error) into a status code and JSON. Here it is in plain Python, so nothing about FastAPI will feel like magic:

Building an API with FastAPI

FastAPI is a modern ASGI framework built on two libraries: Starlette (routing, requests, responses) and Pydantic (data validation from type hints). You declare what a handler needs with ordinary type annotations, and FastAPI parses, validates, converts, documents, and serializes for you.

What FastAPI did for you from those annotations:

• Path parameters (note_id: int) come from {note_id} in the path and are converted; /notes/abc gets an automatic 422 with a precise error message.
• Query parameters are any other simple-typed arguments. Defaults make them optional; Query(ge=1, le=100) adds constraints.
• Body: a Pydantic model parameter is parsed from the JSON body and validated. response_model filters and documents what you return.
• Dependency injection: Depends(get_store) asks FastAPI to call get_store and pass the result. Dependencies can depend on other dependencies (a DB session, the current user, settings), and tests can replace any of them.
• Errors: raising HTTPException produces a JSON error response. For your own domain errors, register a handler with @app.exception_handler(NoteNotFound) so routes stay clean.

Testing uses TestClient, which calls the app in-process (no server, no network) through httpx:

WSGI vs ASGI

How does a server such as uvicorn talk to your framework? Through a standard interface, so any server works with any framework.

• WSGI (PEP 3333, 2003) is synchronous: the app is a callable app(environ, start_response) that receives a dict describing the request and returns an iterable of bytes. Flask and classic Django use it; servers include gunicorn and the standard library's wsgiref. One request occupies one worker thread for its whole life.
• ASGI is its async successor: the app is async def app(scope, receive, send), exchanging event dicts with the server. It supports long-lived connections (WebSockets, server-sent events) and thousands of concurrent slow requests on one process. FastAPI, Starlette, and modern Django use it; servers include uvicorn, hypercorn, and granian.

Both are just functions, so you can call them directly with fake inputs:

A first look at API security

Security deserves its own course; here are the ideas every API author must know on day one (see also M43).

• HTTPS everywhere. Tokens and passwords sent over plain HTTP can be read by anyone on the path.
• Authentication vs authorization. Authentication answers "who are you?" (API keys, OAuth 2.0 bearer tokens, session cookies). Authorization answers "may you do this?", and must be checked on every request for every object: fetching /notes/43 must verify that note 43 belongs to the caller. Missing object-level checks are the number-one API vulnerability in the OWASP API Security Top 10.
• Tokens. JSON Web Tokens (JWTs) are signed, not encrypted; never put secrets in them, always verify the signature and expiry. FastAPI provides OAuth2PasswordBearer and HTTPBearer dependencies to extract tokens.
• CORS (Cross-Origin Resource Sharing). Browsers block a page on https://app.com from reading responses from https://api.com unless the API answers with Access-Control-Allow-Origin. It protects users' browsers, not your server: curl ignores it. In FastAPI, add CORSMiddleware with an explicit list of origins, never "*" together with credentials.
• Validate input, limit everything: body size, page size (le=100), and request rate (429 with Retry-After).
• Secrets in the environment, not in code or logs. Scrub Authorization headers from structured logs.

from fastapi.middleware.cors import CORSMiddleware

app.add_middleware(
    CORSMiddleware,
    allow_origins=["https://notes.example.com"],
    allow_methods=["GET", "POST", "DELETE"],
    allow_headers=["Authorization", "Content-Type"],
)

Practice

These exercises use only the standard library, so they run in the browser and are graded against real Python.

from urllib.parse import urlencode, urljoin

def build_url(base, path, params):
    url = urljoin(base, path)
    kept = {key: value for key, value in params.items() if value is not None}
    return f"{url}?{urlencode(kept)}" if kept else url

CLASSES = {1: "informational", 2: "success", 3: "redirect", 4: "client error", 5: "server error"}
RETRYABLE = {429, 502, 503, 504}
IDEMPOTENT = {"GET", "HEAD", "PUT", "DELETE", "OPTIONS"}

def classify(status):
    if not 100 <= status <= 599:
        raise ValueError(f"not an HTTP status: {status}")
    return CLASSES[status // 100]

def should_retry(method, status):
    return method.upper() in IDEMPOTENT and status in RETRYABLE

class TransientError(Exception):
    pass

def call_with_retry(call, attempts, base, sleep):
    for i in range(attempts):
        try:
            return call()
        except TransientError:
            if i == attempts - 1:
                raise
            sleep(base * 2 ** i)

def fetch_all(get_page, max_pages=100):
    items, cursor = [], None
    for _ in range(max_pages):
        page = get_page(cursor)
        items.extend(page["items"])
        cursor = page["next"]
        if cursor is None:
            return items
    raise RuntimeError(f"gave up after {max_pages} pages")

import re

class Router:
    def __init__(self):
        self.routes = []

    def add(self, method, pattern, handler):
        parts = re.split(r"(\{\w+\})", pattern)
        regex = "".join(
            f"(?P<{part[1:-1]}>[^/]+)" if part.startswith("{") else re.escape(part) for part in parts
        )
        self.routes.append((method.upper(), re.compile(regex), handler))

    def dispatch(self, method, path):
        path_matched = False
        for route_method, regex, handler in self.routes:
            match = regex.fullmatch(path)
            if match is None:
                continue
            path_matched = True
            if route_method == method.upper():
                return handler(**match.groupdict())
        return (405, "method not allowed") if path_matched else (404, "not found")

Mini-project: a notes REST API with FastAPI

Quiz

Recap

• HTTP is stateless request/response: method + path + headers + body in, status + headers + body out.
• Safe methods do not change state; idempotent ones can be retried. POST is neither.
• 4xx means fix the request; 5xx and 429 may be transient. Build URLs with urllib.parse, not string concatenation.
• Robust clients set timeouts, reuse a client (connection pool), read secrets from the environment, follow pagination, and retry transient failures with exponential backoff and jitter.
• Inject the HTTP client or transport so tests use fakes, MockTransport, or respx.
• REST: plural-noun resources, honest status codes, versioning, pagination, one error format.
• FastAPI turns type hints into parsing, validation, docs, and dependency injection; test it with TestClient and dependency_overrides.
• WSGI is sync, ASGI is async; CORS protects browsers, and authorization must be checked per object.

Going further

• MDN: An overview of HTTP and HTTP status codes
• urllib.parse, urllib.request, and http (Python docs)
• HTTPX documentation and RESPX
• FastAPI tutorial and Pydantic
• PEP 3333 (WSGI) and the ASGI specification
• RFC 9457: Problem Details for HTTP APIs and the OWASP API Security Top 10
• Next: M46 · Software Design & Architecture