ML in Production

The ML lifecycle

Industry surveys and experience agree on an uncomfortable fact: the model-fitting code is a small fraction of a production ML system. Around it sit data collection, validation, feature pipelines, serving infrastructure, monitoring, and the processes to retrain safely. The work is a loop: once deployed, the world changes, monitoring notices, and you go around again.

flowchart LR
  A[Frame the problem
metric + baseline] --> B[Collect & validate data]
  B --> C[Features & training
pipelines]
  C --> D[Evaluate
offline + slices]
  D -->|good enough?| E[Package & version]
  D -->|no| C
  E --> F[Deploy
batch or online]
  F --> G[Monitor
data, predictions, outcomes]
  G -->|drift or decay| B
  G -->|business change| A
    

• Frame the problem: what decision does the prediction drive, what metric reflects the business goal, and what simple baseline (a rule, last week's average) must you beat?
• Data: where it comes from, how labels are produced, what is allowed (privacy, consent).
• Train and evaluate: M55–M58, plus the evaluation in this module.
• Package, deploy, monitor: the rest of this module.

Remember: ML in production is a loop of data → model → deployment → monitoring → back to data.

Reproducibility

"It worked on my laptop last month" is not good enough when a regulator, a colleague, or you in six months asks why the model made a decision. A training run is reproducible when the same code, data, environment, and configuration give the same model. Control all four:

• Randomness: pass explicit seeds everywhere: np.random.default_rng(seed), random_state=seed in scikit-learn, torch.manual_seed(seed). Avoid global random state. (Some GPU operations are nondeterministic even with seeds; frameworks document flags for deterministic mode.)
• Environment: pixi.lock pins the exact version and build of every package, Python included, for every platform. Commit it. pixi install --locked fails rather than silently changing versions, which is what you want in CI and in production images.
• Code: the git commit hash of the training code.
• Data: data changes under you. Record a fingerprint (a hash of the file or of a canonical serialization) and keep immutable, dated snapshots. Tools such as DVC or lakeFS version large datasets alongside git; a content hash in your run metadata is the minimum.

Experiment tracking

You will train dozens of variants. An experiment tracker records each run's parameters, metrics, artifacts (plots, the model file), code version, and data fingerprint, and lets you compare runs in a UI. MLflow is a widely used open-source option (others include Weights & Biases and Neptune). Add it to a project with pixi add mlflow; the core API is a few calls:

Even without a tracker, append one JSON line per run (parameters, metrics, data hash, git commit) to a file. That habit alone answers most "which run produced this model?" questions:

Remember: seeds + pixi.lock + git commit + data fingerprint = a reproducible run; track every run.

Evaluation beyond accuracy

Calibration

When a model says "70% chance of default", do 70% of such customers actually default? If so, the model is calibrated. Calibration matters whenever probabilities feed a decision (pricing, triage, expected revenue). A reliability diagram bins predictions by predicted probability and plots the observed frequency in each bin: a calibrated model follows the diagonal. Two summary numbers: the Brier score (mean squared error of probabilities; lower is better) and the expected calibration error (the average gap between confidence and frequency, weighted by bin size). Logistic regression tends to be well calibrated; tree ensembles and naive Bayes often are not. CalibratedClassifierCV fixes a model's probabilities with sigmoid (Platt) or isotonic regression fitted on held-out folds.

Slicing by subgroup

An overall metric can hide a model that works well for most people and badly for some. Slice your evaluation by meaningful groups (region, device, age band, new vs returning customers) and compare. Large gaps are bugs to investigate: often a group is underrepresented in training data, or a feature means something different for it.

Fairness checks

When predictions affect people (loans, hiring, healthcare), check whether outcomes differ across protected groups. Two common, simple metrics compare groups A and B:

• Demographic parity difference: difference in the rate of positive predictions (P(ŷ = 1 | A) − P(ŷ = 1 | B)).
• Equal opportunity difference: difference in recall (true-positive rate) between groups: qualified people should be approved at the same rate regardless of group.

These criteria can conflict with each other and with accuracy, and no metric settles what is fair; that is a decision for people, informed by law and context. Removing the protected attribute from the features does not guarantee fairness, because other features (postcode, for example) can act as proxies. Libraries such as Fairlearn provide these metrics and mitigation methods.

Remember: check calibration when probabilities matter, slice metrics by subgroup, and measure fairness gaps explicitly; decide trade-offs with stakeholders.

Model persistence and versioning

M56 showed how to save a fitted pipeline with joblib or skops. In production you also need to know, for any deployed model: what it is, how it was made, and whether it is safe to load. Save a bundle: the model file plus a metadata file with the model version, training date, data fingerprint, code commit, library versions, input schema, evaluation metrics, and decision threshold. Give models version numbers (for example churn-model 3.2.0, where a major bump means the input schema changed).

A model registry (MLflow's, or a cloud provider's) stores these bundles and tracks their stage: "staging", "production", "archived". Promotion to production should be a reviewed, reversible step, and rollback to the previous version should be one command.

Remember: a model is a file plus metadata; version it, register it, and make rollback easy.

Serving a model: batch vs online

Prefer batch unless the decision truly needs fresh, per-request predictions. For online serving, a small web API is the usual shape. Using FastAPI from M45, the model loads once at startup, a Pydantic model validates every request against the input schema, and each prediction is logged as a structured event with structlog (not print) so it can be monitored later.

Invalid input (a negative tenure, a missing field) is rejected by Pydantic with a 422 response before it reaches the model. The same validation idea, written by hand, is a good exercise and appears in this module's practice.

Remember: batch when you can, online when you must; load once, validate every input, log every prediction as a structured event.

Monitoring, data drift, and concept drift

Deployed models decay because the world changes. Two kinds of change:

• Data drift (covariate shift): the distribution of inputs changes. A marketing campaign brings younger customers; a sensor is recalibrated. The model may now be extrapolating.
• Concept drift: the relationship between inputs and the target changes. Fraudsters adopt a new trick; a pandemic changes what "normal" spending looks like. The same inputs now mean different outcomes.

Concept drift shows up only when true outcomes (labels) arrive, which can take weeks. Data drift can be detected immediately by comparing recent inputs with a reference window (usually the training data). So production monitoring tracks: input distributions per feature, the prediction distribution, missing-value and error rates, latency, and, once labels arrive, the real metric.

Two drift tests

• The two-sample Kolmogorov-Smirnov (KS) test (scipy.stats.ks_2samp) compares two samples of a numeric feature. Its statistic is the largest vertical gap between the two empirical cumulative distribution functions; a small p-value says "these probably come from different distributions". With very large samples, even trivial differences become "significant", so also look at the statistic's size.
• The population stability index (PSI) bins the reference into (say) 10 quantile bins and compares the fraction of reference and current data in each: PSI = Σ (cur − ref) · ln(cur / ref). A common rule of thumb: below 0.1 stable, 0.1–0.25 moderate shift, above 0.25 significant shift. Popular in credit scoring because it is simple to compute and explain.

Remember: data drift = inputs change (detect now with KS/PSI); concept drift = the input-target relationship changes (detect when labels arrive); monitor inputs, outputs, and outcomes.

Testing ML code

ML systems need ordinary software tests (M17) plus tests specific to data and models. Run them with pixi run pytest in CI like any other project.

# tests/unit/test_features.py
import pandas as pd
from churn.features import add_tenure_band

def test_tenure_band_edges():
    df = pd.DataFrame({"tenure_months": [0, 11, 12, 35, 36]})
    out = add_tenure_band(df)
    assert out["tenure_band"].tolist() == ["new", "new", "established", "established", "loyal"]

def test_does_not_mutate_input():
    df = pd.DataFrame({"tenure_months": [5]})
    add_tenure_band(df)
    assert list(df.columns) == ["tenure_months"]

# tests/integration/test_golden.py
import json
from pathlib import Path

import pandas as pd
import pytest
from churn.model import load_bundle

GOLDEN = Path(__file__).parent / "golden"

@pytest.mark.integration
def test_predictions_match_golden_file():
    """A small fixed input set whose expected outputs were reviewed and committed."""
    model = load_bundle("bundle/")
    inputs = pd.read_csv(GOLDEN / "inputs.csv")
    expected = json.loads((GOLDEN / "expected.json").read_text())
    got = model.predict_proba(inputs)[:, 1]
    assert got == pytest.approx(expected["proba"], abs=1e-6)

# tests/integration/test_behavior.py
import pytest
from churn.model import load_bundle

@pytest.mark.integration
def test_more_support_calls_never_lowers_risk(sample_customer):
    model = load_bundle("bundle/")
    low = model.predict_proba(sample_customer(support_calls=0))[0, 1]
    high = model.predict_proba(sample_customer(support_calls=8))[0, 1]
    assert high >= low, "directional expectation agreed with the business"

@pytest.mark.integration
def test_minimum_quality(holdout):
    model = load_bundle("bundle/")
    assert holdout.roc_auc(model) >= 0.80, "release gate: never ship below the agreed floor"

• Unit tests for features: pure functions with small hand-made DataFrames, including edge cases (zero, missing, unseen categories).
• Data validation tests: schema, ranges, null rates, uniqueness of keys, run on every new batch (libraries like pandera or Great Expectations help).
• Golden datasets: a small committed input file with reviewed expected outputs; catches accidental changes to preprocessing or model loading.
• Behavioral tests: invariance (changing a name should not change a credit score), directional expectations, and minimum-performance gates on a frozen holdout.
• Training smoke test: train on 200 rows in CI to prove the pipeline runs end to end.

Remember: test features like functions, data like inputs, and models like behavior; gate releases on a frozen holdout.

Responsible AI

Models make or shape decisions about people. Responsible practice is part of the engineering, not an afterthought:

Remember: define harm, protect data, measure fairness, document with model cards, keep humans in the loop.

Practice

NumPy, SciPy, and pandas are available.

import numpy as np

def psi(reference, current, bins=10):
    edges = np.quantile(reference, np.linspace(0, 1, bins + 1))
    edges[0], edges[-1] = -np.inf, np.inf
    ref = np.clip(np.histogram(reference, edges)[0] / len(reference), 1e-6, None)
    cur = np.clip(np.histogram(current, edges)[0] / len(current), 1e-6, None)
    return float(np.sum((cur - ref) * np.log(cur / ref)))

def validate(record, schema):
    errors = []
    for name, rule in schema.items():
        if name not in record:
            errors.append(f"missing: {name}")
            continue
        value = record[name]
        expected = rule["type"]
        accepted = (int, float) if expected is float else (expected,)
        if isinstance(value, bool) or not isinstance(value, accepted):
            errors.append(f"type: {name}")
            continue
        if "min" in rule and value < rule["min"] or "max" in rule and value > rule["max"]:
            errors.append(f"range: {name}")
        if "allowed" in rule and value not in rule["allowed"]:
            errors.append(f"value: {name}")
    errors.extend(f"unexpected: {name}" for name in sorted(set(record) - set(schema)))
    return errors

import numpy as np

def metrics_by_group(y_true, y_pred, groups):
    t, p, g = np.asarray(y_true), np.asarray(y_pred), np.asarray(groups)
    result = {}
    for value in np.unique(g):
        m = g == value
        positives = t[m] == 1
        result[value.item()] = {
            "n": int(m.sum()),
            "accuracy": float(np.mean(t[m] == p[m])),
            "recall": float(np.mean(p[m][positives] == 1)) if positives.any() else 0.0,
            "positive_rate": float(np.mean(p[m] == 1)),
        }
    return result

def max_gap(by_group, metric):
    values = [stats[metric] for stats in by_group.values()]
    return max(values) - min(values)

from scipy.stats import ks_2samp

def drift_report(reference, current, alpha=0.01):
    report = {}
    for name, ref_values in reference.items():
        result = ks_2samp(ref_values, current[name])
        p_value = float(result.pvalue)
        report[name] = {"statistic": float(result.statistic), "p_value": p_value, "drifted": bool(p_value < alpha)}
    return report

import numpy as np

def ece(y_true, proba, n_bins=10):
    y, p = np.asarray(y_true, dtype=float), np.asarray(proba, dtype=float)
    idx = np.minimum((p * n_bins).astype(int), n_bins - 1)
    total = 0.0
    for b in range(n_bins):
        m = idx == b
        if m.any():
            total += m.mean() * abs(p[m].mean() - y[m].mean())
    return float(total)

Mini-project

Quiz

Recap

• Production ML is a loop: frame → data → train → evaluate → package → deploy → monitor → back again.
• Reproducibility needs seeds, pixi.lock, the code commit, and a data fingerprint; track every run (MLflow or a JSON log).
• Evaluate calibration (reliability diagram, Brier, ECE), slice metrics by subgroup, and measure fairness gaps.
• Persist models as bundles with metadata and hashes; version and register them; make rollback easy.
• Prefer batch prediction; for online serving load once, validate inputs, and log structured events.
• Detect data drift with KS and PSI; concept drift appears when labels arrive.
• Test features, data, and model behavior; gate releases on a frozen holdout and golden files.
• Practice responsible AI: define harm, protect data, check fairness, document, keep humans in the loop.

Going further

• scikit-learn: probability calibration
• scikit-learn: model persistence
• scipy.stats.ks_2samp
• MLflow documentation
• FastAPI documentation
• Fairlearn (fairness metrics and mitigation)
• Pixi documentation (lock files and --locked)
• Next: put it all together in Project 8: Movie Recommender, then continue to M60 · Data Engineering Foundations.