Part 8 · Machine Learning · Graded project

Project 8: Movie Recommender

Build the recommendation engine for a small streaming service, from a sparse ratings matrix to a tested recommend(user_id, k) API. You will implement five model families, measure them with the offline metrics real teams use, and show that matrix factorization finds hidden tastes a popularity chart cannot. Fifty-one tests tell you when each piece is right.

⏱ ~10–14 hoursPrerequisites: M55, M56, M57, M59

The scenario

Reelhouse is a small streaming service: a few hundred subscribers, a few hundred films. Today the home page shows the same "Top 10" to everyone, and people stop browsing after the first row. The product lead's request is short: "Show each subscriber films they will like, and prove it works before we ship it."

That sentence hides three jobs. First, a model that predicts how much a user would like a film they have not seen. Second, an evaluation that shows, without real users, that the model beats what we have now. Third, an API the web team can call with a user id. You will do all three.

Real ratings are private, so the package ships a deterministic synthetic generator, generate_ratings(n_users, n_items, seed). It is not random noise: every user and every film has a hidden three-number "taste" vector (think "how much action, how much drama, how much sci-fi"), plus a bias (some users are generous, some films are just good). A rating is roughly 3 + user_bias + film_bias + 1.2 · (taste_user · taste_film) / √3, plus noise, rounded to whole stars. Genres are derived from the same hidden tastes. So there is real structure to find, which means your evaluation can tell good models from bad ones.

Learning goals

Sparse data

Most users rate a tiny fraction of the catalogue. CSR matrices store only what exists and make "all ratings of user u" a cheap slice.

Collaborative filtering

"People who liked what you liked also liked…" expressed as cosine similarity between rating vectors.

Latent factors

Matrix factorization learns the hidden taste vectors directly, with gradient descent and regularization from M55.

Honest evaluation

Rating error (RMSE) and ranking quality (precision@k, recall@k) can disagree. Coverage catches models that only ever recommend hits.

Feature specification

Every model subclasses Recommender (provided in base.py):

class Recommender(ABC):
    def fit(self, train: RatingMatrix) -> Self: ...            # learn; store train in self._train
    def score_items(self, user: int) -> np.ndarray: ...       # predicted rating for EVERY item (1-5 scale)
    def predict(self, user: int, item: int) -> float: ...     # provided: score_items(user)[item]
    def predict_many(self, pairs: np.ndarray) -> np.ndarray:  # provided: batch predictions for RMSE

Because every model returns predicted ratings, the same scores serve both RMSE (how close is the predicted star count?) and ranking (sort the scores, take the top k).

ModulePublic APIBehavior
datagenerate_ratings, generate_items (provided), load_ratings_csv, build_matrix, RatingMatrix Sorted id → index maps; 0 means unrated so ratings must be positive; duplicates raise ValueError; optional item_ids adds cold items.
baselinebayesian_average, PopularityRecommender(prior_weight=5) (C·m + Σr) / (C + n) per item, where m is the global mean.
neighborscosine_similarity, top_k_mask, ItemKNN(k), UserKNN(k) Zero vectors have similarity 0 (never NaN). Predictions are a mean plus a similarity-weighted average of deviations; no neighbors means fall back to the mean.
mfrmse, MatrixFactorization(n_factors, lr, reg, n_epochs, seed) r̂ = μ + b_u + b_i + p_u·q_i; seeded SGD; history_ holds training RMSE per epoch; predictions clipped to 1–5.
contentgenre_matrix, ContentRecommender(features, k), HybridRecommender([(model, w), …]) Genre cosine replaces rating cosine, so never-rated films still get scores. The hybrid normalizes weights to sum to 1.
evaluatetrain_test_split_by_user, top_k, precision_at_k, recall_at_k, coverage, evaluate, compare_models, format_table Each user with enough ratings gives some to test and keeps at least one in train. Rankings exclude training items. Ties break toward the lower index.
apiRecommenderService(model).fit(data).recommend(user_id, k=10, exclude_seen=True) Returns Recommendation(item_id, score) objects, best first. Unknown users get the popularity ranking.

Architecture

flowchart LR
  G[generate_ratings / CSV] --> B[build_matrix]
  B --> M[(RatingMatrix: CSR + id maps)]
  M --> S[train_test_split_by_user]
  S --> T[train matrix]
  S --> H[held-out ratings]
  T --> P[PopularityRecommender]
  T --> K[ItemKNN / UserKNN]
  T --> F[MatrixFactorization]
  T --> C[ContentRecommender]
  I[generate_items: genres] --> C
  F --> Y[HybridRecommender]
  C --> Y
  P --> E[evaluate / compare_models]
  K --> E
  F --> E
  Y --> E
  H --> E
  E --> R[format_table]
  F --> A[RecommenderService.recommend]
  P -. cold users .-> A

The data layer knows nothing about models; models know only RatingMatrix; evaluation knows only the Recommender interface. That is why one evaluate function can compare every model, and why the API can serve any of them.

Try the two core data ideas right here in the browser (the first SciPy import takes a few seconds):

Getting started with Pixi

pixi run projects                  # list projects and your progress
pixi run start p08                 # copy the starter into work/p08_recommender/
pixi run grade p08                 # run the 51 tests against your work/ copy
pixi run grade p08 -- -k recommender_data   # only the milestone 1 tests (pytest -k)
cd work/p08_recommender
pixi run python -m recommender     # print the model comparison table once everything works

The starter gives you the complete generator and the Recommender base class. Every other function has its real signature and docstring, a body of raise NotImplementedError, and # TODO comments describing the approach. Work milestone by milestone; the tests are grouped the same way.

Milestones

Milestone 1 · Data loading and the sparse matrix

Implement load_ratings_csv, build_matrix, and the RatingMatrix helpers. Users and items are indexed in sorted id order so the same data always gives the same matrix.

  • build_matrix returns a csr_matrix of shape (n_users, n_items) with user_index/item_index maps
  • Duplicate (user, item) pairs and zero/negative ratings raise ValueError
  • Passing item_ids adds cold catalogue items as empty columns
  • user_ratings(u) slices indices/data with indptr (no dense conversion)
  • pixi run grade p08 -- -k recommender_data passes

Milestone 2 · Popularity baseline

Implement bayesian_average and PopularityRecommender. This is the number every other model must beat.

  • The damped mean matches (C·m + Σr) / (C + n) for hand-computed inputs
  • An item with no ratings scores exactly the global mean (no division by zero)
  • Four good ratings outrank one perfect rating

Milestone 3 · Item-item and user-user collaborative filtering

Implement cosine_similarity (sparse-friendly, zero rows give 0), top_k_mask, ItemKNN, and UserKNN. Both predict mean + Σ sim·deviation / Σ sim over the k most similar neighbors with positive similarity.

  • Item-item similarity is computed once in fit from train.matrix.T
  • A cold item (or one with no similar rated items) predicts the user's mean
  • User-user kNN uses each neighbor's deviation from their own mean
  • The small hand-computed examples in the tests give 5.0, 2.0, and 1.0

Milestone 4 · Matrix factorization

Implement rmse and MatrixFactorization.fit/score_items. This is gradient descent from M55 on a new loss: squared error over the observed ratings plus reg · (b_u² + b_i² + |p_u|² + |q_i|²).

  • Initialization and shuffling use np.random.default_rng(self.seed), so runs are reproducible
  • history_ gets one training RMSE per epoch and it goes down
  • Optional validation rows fill validation_history_
  • Predictions are clipped to self.clip (1–5)
  • The statistical test passes: MF beats popularity by at least 0.1 RMSE

Milestone 5 · Content-based and hybrid

Implement genre_matrix, ContentRecommender, and HybridRecommender. Content similarity does not need ratings, so it can recommend a film on the day it is added.

  • Genre vectors follow the matrix's item order; unknown genres and missing metadata give zero rows
  • A cold item that shares a genre with a loved film gets a high score
  • The hybrid validates weights and returns Σ w·score / Σ w

Milestone 6 · Evaluation and the comparison table

Implement the split, the metrics, evaluate, compare_models, and format_table. Then run pixi run python -m recommender and read your table.

  • The split holds out ratings per user; every eligible user keeps at least one training rating
  • Rankings exclude items the user rated in training
  • precision@k divides by k; recall@k divides by the number of relevant items
  • Coverage is the fraction of the catalogue that appears in any user's top-k list
  • You can explain in two sentences why your best RMSE model and best coverage model differ

Milestone 7 · The recommend() API

Implement RecommenderService.fit and recommend. This is the only function the web team will ever call, so its contract matters more than its internals.

  • recommend(user_id, k, exclude_seen=True) returns at most k Recommendations, best first
  • exclude_seen=False allows already-rated items
  • Unknown users get the popularity ranking instead of an exception
  • k < 1 raises ValueError; calling before fit raises RuntimeError
  • pixi run grade p08 reports 51 passed

Hints

Building the CSR matrix

csr_matrix((values, (rows, cols)), shape=(n_users, n_items)) builds it in one call from three aligned lists. Map ids to positions with dicts built from the sorted unique ids. Call matrix.sort_indices() so each row's columns are in ascending order; the tests rely on it.

Cosine similarity without dividing by zero 
norms = np.sqrt(np.asarray(X.multiply(X).sum(axis=1)).ravel())
inverse = np.divide(1.0, norms, out=np.zeros_like(norms), where=norms > 0)
unit = scipy.sparse.diags(inverse) @ X          # each nonzero row now has length 1
sim = (unit @ unit.T).toarray()

A zero row stays zero after scaling, so all its similarities are 0 rather than NaN.

Vectorizing item-kNN for one user

Take the user's rated item indices J and ratings r. Then W = top_k_mask(sim[:, J], k) is an (n_items, |J|) weight matrix, and every prediction at once is mean + (W @ (r - mean)) / W.sum(axis=1), using np.where where the denominator is 0.

The SGD update for matrix factorization

For one rating, err = r - (mu + b_u + b_i + p_u @ q_i). The gradient of err² + reg·(…) gives these updates (the factor 2 is folded into the learning rate):

b_u += lr * (err - reg * b_u)
b_i += lr * (err - reg * b_i)
p_u, q_i = p_u + lr * (err * q_i - reg * p_u), q_i + lr * (err * p_u - reg * q_i)

Update p_u and q_i from the old values (the tuple assignment does that). If RMSE explodes to NaN, your learning rate is too high.

My MF does not beat the baseline

Check that you shuffle each epoch, that biases start at zero and factors at small random values (not zero: with all-zero factors the gradients for p and q are zero forever), and that the test uses enough epochs. With 3 factors, lr=0.03, 30 epochs, the reference solution gets test RMSE around 0.7–0.85 versus about 1.1–1.2 for popularity.

Deterministic ties in top_k

np.argsort(-scores, kind="stable") keeps equal scores in index order. Set excluded items to -np.inf first, then drop them from the result.

Rubric

AreaWeightWhat "excellent" looks like
Correctness40%All 51 tests pass, including the statistical MF-vs-baseline check.
Data handling10%Sparse operations throughout; no full dense user × item matrix except where clearly small.
Evaluation20%A comparison table with all models and a short written interpretation (which model wins which metric, and why).
Code quality15%Type hints, Google docstrings, ruff-clean, small functions, no duplicated scoring logic.
API design10%Clear errors, cold-user fallback, deterministic output.
Reproducibility5%Seeds everywhere randomness appears; the same command prints the same table.

Stretch goals

Reflection quiz