Project 5: Async Web Crawler

The scenario

Your team's documentation site has grown to hundreds of pages written by dozens of people. Pages get renamed, sections move, and readers keep landing on 404s. You are asked for a tool that can run in CI and answer one question: which links are broken, and where are they? A plain "this URL is a 404" is not enough. The writer fixing it needs the referrers, the pages that contain the bad link.

Fetching pages one at a time is far too slow: a 300-page site at 200 ms per request takes a minute, and most of that time the program sits idle waiting on the network. That is exactly the I/O-bound workload asyncio was built for (M38). But unbounded concurrency would hammer the server, so the crawler must also be polite: obey robots.txt, cap concurrent requests, and space out requests per host.

Learning goals

Feature specification

Architecture

The crawl is a classic producer/consumer pipeline where the consumers are also producers: each worker pulls a Job(url, depth, referrer), fetches it, and pushes newly discovered links back onto the same queue. The queue's unfinished-task counter is what tells us we are done: queue.join() returns only when every job that was ever put has had a matching task_done().

flowchart LR
  S[start URL] --> Q[(asyncio.Queue of Job)]
  subgraph TG[TaskGroup]
    W1[worker 1]
    W2[worker 2]
    W3[worker N]
  end
  Q --> W1 & W2 & W3
  W1 & W2 & W3 --> R{robots policy
cached per host}
  R -->|allowed| L[HostRateLimiter]
  L --> F[fetch_with_retries]
  F --> FE[[Fetcher Protocol
FakeFetcher / UrllibFetcher]]
  F --> P[extract_links + normalize_url]
  P -->|same site, new, depth ok| Q
  P -->|off site| X[external set]
  W1 & W2 & W3 --> REP[results, referrers, redirects]
  REP --> J[CrawlReport.to_json]
    

Getting started with Pixi

pixi run projects            # list projects and which you've started
pixi run start p05           # copy the starter into work/p05_async_crawler/
pixi run grade p05           # run the 81 grading tests against your work/ copy
pixi run practicum grade p05 -k urls -x   # one milestone at a time, stop at first failure

After the tests pass, try it for real on a site you control:

cd work/p05_async_crawler
pixi run python -m crawler https://docs.example.com/ --max-depth 2 --interval 1.0 > report.json

Milestones

The heart of it: worker pool and shutdown

This is the shape the tests expect. Study why each try/finally is there before you write yours.

async def crawl(self, start_url: str) -> CrawlReport:
    start = normalize_url(start_url)
    if start is None:
        raise ValueError(f"not a crawlable URL: {start_url!r}")
    self._reset(start)
    queue: asyncio.Queue[Job | None] = asyncio.Queue()
    self._schedule(queue, Job(start, 0, None))
    async with asyncio.TaskGroup() as group:
        for i in range(self.config.concurrency):
            group.create_task(self._worker(queue), name=f"crawler-worker-{i}")
        await queue.join()                  # every put has a matching task_done
        for _ in range(self.config.concurrency):
            queue.put_nowait(None)          # one sentinel per worker
    return self.report()

async def _worker(self, queue):
    while True:
        job = await queue.get()
        try:
            if job is None:
                return
            ...                             # robots, rate limit, fetch, parse, schedule
        finally:
            queue.task_done()               # even on error or cancellation

Rubric

Hints

Stretch goals

• Check (but do not crawl) external links, with their own per-host limits.
• Honor Retry-After on 429/503 and <meta name="robots" content="nofollow">.
• Write an httpx.AsyncClient fetcher (pixi add httpx) and benchmark it against the thread-backed one.
• Persist the frontier in sqlite3 so an interrupted crawl can resume.
• While it runs, inspect it with python -m asyncio pstree PID (3.14) to see the worker tree.
• Emit progress with structlog (previewed in M42).

Reflection quiz