Machine Learning Foundations

What machine learning is (and is not)

In ordinary programming you write the rules: "if the email contains 'WIN A PRIZE', mark it as spam". In machine learning (ML) you supply examples (thousands of emails already marked spam or not) and an algorithm finds the rules for you. More precisely, an ML algorithm picks a function f from a family of candidate functions (the model) so that f(input) is close to the desired output on the examples, and, crucially, on new examples it has never seen. That second part, generalization, is the whole point. Memorizing the training emails is easy; correctly flagging tomorrow's spam is the job.

Three ingredients appear in every ML method, from a straight line to a large language model:

1. A model family: the shapes of function allowed. "All straight lines y = w·x + b" is a family; each choice of the parameters w and b is one member.
2. A loss function: a number that says how wrong a particular member is on the data. Smaller is better.
3. An optimizer: a procedure that searches the family for parameters with small loss, often gradient descent.

ML is not a replacement for thinking. It cannot learn a pattern that is not in the data, it inherits every bias in how the data was collected, and it gives confident wrong answers outside the range of its examples. If a simple rule works ("orders over $10,000 need manual review"), write the rule. Reach for ML when the rules are too many, too fuzzy, or too fast-changing to write by hand: recognizing handwriting, ranking search results, predicting demand.

Remember: ML = model family + loss + optimizer, judged by performance on unseen data.

Features, labels, and the feature matrix

Each example (also called a sample, row, or observation) is described by a fixed list of numbers called features. A house might have features [area_m2, bedrooms, age_years]. Stack the feature lists of all n examples as rows and you get the feature matrix X with shape (n_samples, n_features). The answers form the label vector (or target) y with shape (n_samples,). Row i of X goes with entry i of y.

Every scikit-learn estimator, and every from-scratch model in this module, expects exactly this layout. A single feature still needs a 2-D X: shape (n, 1), not (n,). Text, images, and categories must first be turned into numbers (feature engineering); M56 shows the standard tools.

Remember: X is (n_samples, n_features), y is (n_samples,), and row i of one matches entry i of the other.

Train, validation, and test splits (and leakage)

A model that scores perfectly on the data it learned from may have simply memorized it. To estimate how it will do on future data we hold some data back and never let the model see it while learning. The standard arrangement uses three parts:

• Training set (often 60–80%): the model's parameters are fitted on it.
• Validation set (10–20%): used to make choices: which model, which polynomial degree, which regularization strength (these choices are called hyperparameters, because they are set by you rather than learned from the data).
• Test set (10–20%): used once, at the very end, to report an honest estimate. If you look at the test score and then change the model, the test set has become a second validation set and its score is no longer honest.

Data leakage happens when information from the validation or test data sneaks into training. The result is a model that looks great in the notebook and fails in production. Classic leaks:

• Computing a scaling mean/standard deviation (or filling missing values) on the whole dataset before splitting. The training step has then "seen" test statistics. Fit preprocessing on train only, then apply it to the others. M56's Pipelines automate this.
• Random splits of time series: training on Wednesday to predict Tuesday. Split by time instead.
• Duplicates or the same customer in both train and test. Split by group (customer id), not by row.
• A feature that is a disguised label: "refund_issued" when predicting "complaint filed".

Remember: fit on train, choose on validation, report on test once. Anything computed from data (means, vocabularies, imputation values) is part of fitting and must use train only.

Loss functions: measuring wrongness

A loss function turns predictions and true labels into one number. Training means "find parameters that make this number small", so the loss you choose defines what "good" means.

Mean squared error (regression)

For true values yi and predictions ŷi ("y-hat"), the mean squared error is

MSE = (1/n) Σ (yi − ŷi)²

Squaring makes every error positive and punishes big misses much more than small ones: an error of 4 costs 16, four errors of 1 cost only 4. That is good when big misses are truly worse, and bad when the data has wild outliers (one outlier can dominate the fit). Its square root, RMSE, is in the same units as y, so it is easier to read ("off by about $23k").

Cross-entropy (classification)

A classifier usually outputs a probability p that the label is 1. For a true label y ∈ {0, 1}, the binary cross-entropy (also called log loss) is

L = −[ y·log(p) + (1 − y)·log(1 − p) ]

Read it case by case. If y = 1, the loss is −log(p): 0 when p = 1, about 0.69 when p = 0.5, and enormous as p → 0. A confident wrong answer is punished far harder than a hesitant one. That is exactly the behavior we want from a probability, and it is why we do not use MSE for classification.

Remember: MSE for numbers, cross-entropy for probabilities. The loss encodes what you care about; choosing it is a modeling decision.

Linear regression, part 1: the normal equation

Linear regression predicts a number as a weighted sum of the features plus a constant:

ŷ = w1x1 + w2x2 + … + wdxd + b

The weights w say how much each feature pushes the prediction up or down; the bias (intercept) b is the prediction when every feature is 0. A neat trick folds b into w: add a column of ones to X. Then for all samples at once, ŷ = X w (a matrix-vector product from M49).

Which w minimizes MSE? Calculus gives an exact answer. Setting the derivative of ‖Xw − y‖² to zero yields the normal equation:

(XᵀX) w = Xᵀy

Solve that linear system and you are done, no iteration needed. In NumPy, prefer np.linalg.lstsq(X, y) (or np.linalg.solve(X.T @ X, X.T @ y)) over computing an explicit inverse: it is faster and numerically safer.

Remember: linear regression is ŷ = Xw; least squares has an exact solution, the normal equation, which lstsq computes stably.

Linear regression, part 2: gradient descent

Picture the loss as a landscape: each point on the ground is a choice of parameters, and the height is the loss. You are standing somewhere on a foggy hillside and want to reach the valley floor. You cannot see far, but you can feel which way the ground slopes under your feet. So you take a step downhill, feel again, and repeat. That is gradient descent.

The gradient ∇L(w) is the vector of partial derivatives: for each parameter, "if I nudge this up a little, how fast does the loss go up?" It points uphill, so we step the opposite way:

w ← w − η · ∇L(w)

The step size η (eta) is the learning rate. Try it on the landscape below: drag the learning-rate slider and watch small rates crawl, good rates glide in, and large ones overshoot and diverge.

Deriving the MSE gradient intuitively

For one sample, the error is e = ŷ − y = w·x − y and the loss is e². By the chain rule, the derivative of e² with respect to wj is 2e · xj. Read that as a sentence: the blame assigned to weight j is the error times how much feature j was present. If the prediction was too high (e > 0) and xj was large, wj gets pushed down hard; if xj = 0, that weight was not involved and gets no blame. Averaging over all samples and writing it with matrices gives

∇MSE(w) = (2/n) · Xᵀ (Xw − y)

Xw − y is the vector of errors; multiplying by Xᵀ computes, for every feature at once, "sum over samples of error × feature value". That one line is the engine of linear regression, and the same pattern (error times input) reappears in logistic regression and in every layer of a neural network (M58).

Learning rate and convergence

The algorithm has converged when the loss stops changing meaningfully (or the gradient is nearly zero). Too small a learning rate wastes thousands of steps; too large a rate makes each step overshoot the valley so the loss grows, often to inf or nan. A useful habit: always plot the loss curve. A smooth decreasing curve that flattens is healthy; a zig-zag means the rate is near the edge; a curve that rises means divergence.

Feature scaling

If one feature is measured in square metres (values near 100) and another in bedrooms (values near 2), the loss landscape is a long, narrow valley: very steep along one weight, nearly flat along the other. One learning rate cannot suit both directions, so descent zig-zags across the steep walls while crawling along the floor. Try the elongated bowl below, then imagine it rounded.

Standardization fixes this: subtract each feature's (training) mean and divide by its (training) standard deviation, so every feature has mean 0 and standard deviation 1. The valley becomes round and gradient descent heads straight for the bottom. Scaling does not change what the best model can predict; it changes how fast you find it. (Tree models in M56 do not care about scaling; distance-based models such as k-nearest neighbors care a lot.)

Remember: w ← w − η∇L; the MSE gradient is (2/n)Xᵀ(Xw − y), "error times input"; plot the loss curve; standardize features before gradient descent.

Underfitting, overfitting, and bias vs variance

Give linear regression polynomial features [x, x², …, xd] and it can fit curves. The degree d controls the model's capacity (how wiggly a function it can represent). Too little capacity and the model underfits: it misses real structure and is wrong on train and test. Too much and it overfits: it bends to chase the noise in the training points, so training error is tiny while test error explodes.

Sweep the degree and plot both errors and you get the most important picture in ML: training error falls forever, validation error falls and then rises. The best model sits at the bottom of the validation curve.

The bias-variance trade-off names the two failure modes. Bias is error from wrong assumptions: a straight line can never fit a sine wave, no matter how much data you give it. Variance is error from sensitivity to the particular training sample: refit a degree-15 polynomial on a fresh sample of 20 points and you get a completely different wiggly curve. Simple models have high bias and low variance; flexible models have low bias and high variance. More data reduces variance (it is harder to chase noise in 10,000 points than in 20), which is why big flexible models need big datasets.

Remember: judge models by validation error; the gap between train and validation error is your overfitting meter.

Regularization: ridge and lasso

Overfit polynomials have enormous, alternating weights (+3,000, −9,000, +12,000…) that cancel out on the training points and swing wildly between them. Regularization adds a penalty on weight size to the loss, so the optimizer must trade fit against simplicity:

• Ridge (L2): loss = MSE + α · Σ wj². Shrinks every weight smoothly toward zero. Closed form: w = (XᵀX + αI)⁻¹ Xᵀy. Adding αI also makes the system solvable when features are highly correlated.
• Lasso (L1): loss = MSE + α · Σ |wj|. Pushes many weights to exactly zero, so it doubles as automatic feature selection. No closed form; solved iteratively.

Geometrically, ridge's penalty is a circle and lasso's is a diamond. The optimum is where the loss contours first touch the penalty shape, and a diamond is usually touched at a corner, where some coordinates are zero. The strength α is a hyperparameter: choose it on validation data. Conventionally the bias term is not penalized, and features are standardized first so one penalty is fair to all of them.

Remember: regularization = loss + penalty on weight size. Ridge shrinks, lasso shrinks and zeroes. Tune α on validation data.

Logistic regression from scratch

For classification we want a probability between 0 and 1, but Xw can be any real number. The sigmoid (logistic) function squashes it:

σ(z) = 1 / (1 + e−z)

σ(0) = 0.5, large positive z gives nearly 1, large negative gives nearly 0. Logistic regression predicts p = σ(Xw) and is trained by minimizing cross-entropy. Despite the name it is a classification method. The quantity z = Xw is called the logit (log-odds): z = log(p / (1 − p)).

The gradient is delightfully familiar. Working through the chain rule, the sigmoid's derivative σ(z)(1 − σ(z)) cancels against the derivative of the log, leaving

∇CE(w) = (1/n) · Xᵀ (σ(Xw) − y)

Same shape as linear regression: error (predicted probability minus label) times input. To classify, threshold the probability, usually at 0.5. The set of points where p = 0.5, that is where Xw = 0, is the decision boundary; for logistic regression it is a straight line (a plane in higher dimensions).

Remember: logistic regression = linear score → sigmoid → probability, trained with cross-entropy; its gradient is again "error times input"; its decision boundary is linear.

Evaluation metrics

Accuracy (fraction correct) is the obvious metric and often the wrong one. If 1% of transactions are fraud, a model that always says "not fraud" is 99% accurate and completely useless. Better metrics start from the confusion matrix, which counts the four outcomes of a binary prediction:

• Precision = TP / (TP + FP): of everything we flagged, how much was really positive? High precision means few false alarms.
• Recall (sensitivity) = TP / (TP + FN): of all real positives, how many did we catch? High recall means few misses.
• F1 = 2 · precision · recall / (precision + recall): their harmonic mean, high only when both are high.
• ROC-AUC: lowering the threshold catches more positives (recall up) but raises false alarms. The ROC curve plots true-positive rate against false-positive rate for every threshold; the area under it (AUC) is the probability that a random positive gets a higher score than a random negative. 0.5 is coin-flipping, 1.0 is perfect, and it does not depend on any single threshold.

Which to optimize is a business decision. A cancer screen wants high recall (do not miss cases; a follow-up test handles false alarms). A spam filter wants high precision (never bin a real email).

Regression metrics

• MAE (mean absolute error) = mean of |y − ŷ|: average miss in the target's units; robust to outliers.
• RMSE = √MSE: also in target units, but weights big misses more.
• R² (coefficient of determination) = 1 − SS_res/SS_tot: the fraction of the target's variance explained. 1 is perfect; 0 means "no better than always predicting the mean"; negative means worse than that.

Remember: look at the confusion matrix first; choose precision vs recall by the cost of each mistake; AUC compares rankings independent of threshold; report RMSE/MAE in target units and R² for "how much is explained".

Practice

Each exercise runs your code, then the tests. NumPy is available.

import numpy as np

def split_indices(n, test_fraction, seed):
    order = np.random.default_rng(seed).permutation(n)
    k = int(n * test_fraction)
    return order[k:], order[:k]

import numpy as np

def mse(X, y, w):
    return float(np.mean((X @ w - y) ** 2))

def mse_grad(X, y, w):
    return 2 * X.T @ (X @ w - y) / len(y)

import numpy as np

def fit_gd(X, y, lr=0.1, n_iter=1000):
    Xb = np.column_stack([X, np.ones(len(y))])
    theta = np.zeros(Xb.shape[1])
    losses = []
    for _ in range(n_iter):
        residual = Xb @ theta - y
        theta = theta - lr * 2 * Xb.T @ residual / len(y)
        losses.append(float(np.mean((Xb @ theta - y) ** 2)))
    return theta[:-1], float(theta[-1]), losses

import numpy as np

def classification_report(y_true, y_pred):
    t, p = np.asarray(y_true), np.asarray(y_pred)
    tp = int(np.sum((t == 1) & (p == 1)))
    fp = int(np.sum((t == 0) & (p == 1)))
    fn = int(np.sum((t == 1) & (p == 0)))
    tn = int(np.sum((t == 0) & (p == 0)))
    precision = tp / (tp + fp) if tp + fp else 0.0
    recall = tp / (tp + fn) if tp + fn else 0.0
    f1 = 2 * precision * recall / (precision + recall) if precision + recall else 0.0
    return {"tp": tp, "fp": fp, "fn": fn, "tn": tn, "precision": precision, "recall": recall, "f1": f1}

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-np.asarray(z, dtype=float)))

def log_loss(y, p):
    p = np.clip(np.asarray(p, dtype=float), 1e-12, 1 - 1e-12)
    y = np.asarray(y, dtype=float)
    return float(-np.mean(y * np.log(p) + (1 - y) * np.log(1 - p)))

def fit_logistic(X, y, lr=0.5, n_iter=2000):
    Xb = np.column_stack([X, np.ones(len(y))])
    w = np.zeros(Xb.shape[1])
    for _ in range(n_iter):
        w = w - lr * Xb.T @ (sigmoid(Xb @ w) - y) / len(y)
    return w

Mini-project

Quiz

Recap

• ML picks a function from a model family by minimizing a loss with an optimizer; success means generalizing to new data.
• Supervised learning has labels (regression or classification); unsupervised finds structure; reinforcement learns from rewards.
• X is (n_samples, n_features); y is (n_samples,).
• Fit on train, tune on validation, report on test once; fit every preprocessing statistic on train only.
• MSE for regression, cross-entropy for probabilities.
• Linear regression: exact via the normal equation (lstsq), or iterative via gradient descent with gradient (2/n)Xᵀ(Xw − y).
• The learning rate trades speed against stability; standardized features make descent much faster.
• Underfitting = high bias, overfitting = high variance; regularization (ridge, lasso) controls capacity.
• Logistic regression = sigmoid of a linear score, trained with cross-entropy; its boundary is linear.
• Evaluate classifiers with the confusion matrix, precision, recall, F1, and ROC-AUC; regressors with MAE, RMSE, and R².

Going further

• scikit-learn user guide: linear models
• scikit-learn: metrics and scoring
• scikit-learn: common pitfalls and recommended practices (leakage)
• numpy.linalg.lstsq
• Next: M56 · scikit-learn in Practice turns everything here into reusable, leak-proof pipelines.