scikit-learn in Practice

The estimator API

scikit-learn's great idea is that every algorithm is an object with the same few methods. Learn the pattern once and you can use any of its hundreds of classes:

• An estimator is any object with fit(X, y) (or fit(X) for unsupervised methods). fit learns from data and returns self, so calls can be chained.
• A predictor adds predict(X) (and for classifiers often predict_proba(X) and score(X, y)).
• A transformer adds transform(X), which returns a new feature matrix, and the shortcut fit_transform(X). Scalers, encoders, and imputers are transformers.

Two kinds of numbers live on an estimator, and the naming convention tells them apart. Hyperparameters are settings you choose, passed to the constructor: LogisticRegression(C=0.5). Learned attributes are computed by fit and end in an underscore: coef_, intercept_, classes_, mean_. If an attribute ends in _ and you have not called fit, it does not exist yet.

Remember: fit learns, predict/transform apply. Constructor arguments are hyperparameters; trailing-underscore attributes are learned.

Preprocessing mixed tabular data

Real tables mix numbers ("age", "monthly_spend"), categories ("plan", "country"), and holes (missing values). Most models need a purely numeric matrix with no missing values, so we transform each column type differently:

• SimpleImputer(strategy="median") fills missing numbers with the training median (robust to outliers); strategy="most_frequent" works for categories.
• StandardScaler rescales numbers to mean 0, standard deviation 1 (M55's standardization).
• OneHotEncoder turns a category with k values into k 0/1 columns. handle_unknown="ignore" makes a category never seen in training encode as all zeros instead of crashing in production.
• ColumnTransformer applies a different transformer to each group of columns and concatenates the results side by side.

Remember: impute, then scale numbers; impute, then one-hot categories; glue them with ColumnTransformer; set handle_unknown="ignore" for production safety.

Pipelines prevent leakage

A Pipeline chains transformers and a final estimator into one estimator. Calling pipe.fit(X_train, y_train) fits each step on the training data in order; calling pipe.predict(X_test) only applies the already-fitted steps. That ordering is exactly the "statistics from train only" rule of M55, enforced by code instead of discipline.

The danger is subtle, so here is a dramatic demonstration. We generate pure noise: 60 samples, 2,000 random features, random labels. There is nothing to learn; honest accuracy is 50%. If we select the 20 "best" features on all the data and then cross-validate, the selection step has already peeked at every test fold's labels.

Remember: put every data-dependent step inside the Pipeline, and cross-validate the Pipeline, not pre-transformed data.

A tour of the models

A decision tree is the easiest model to look inside. Each box shows the question, the impurity (Gini: the chance two random samples in the box have different labels), how many samples reach it, and the class counts.

Different models draw different decision boundaries. On the curved make_moons data, compare:

Remember: start with a linear baseline, then try a random forest and a histogram gradient booster; scale features for linear models, k-NN, and SVMs.

Cross-validation

A single validation split is noisy: with 150 samples, a lucky split can swing accuracy by several points. k-fold cross-validation splits the data into k folds, trains k times, each time holding out a different fold, and averages the k scores. Every sample is used for validation exactly once, and the spread of the scores tells you how stable the estimate is.

• KFold(n_splits=5, shuffle=True, random_state=0): plain folds. Use for regression.
• StratifiedKFold: each fold keeps the class proportions. The default for classifiers when you pass cv=5; essential with imbalanced classes.
• GroupKFold keeps all rows of one group (customer, patient) in the same fold; TimeSeriesSplit always validates on data later than the training data.

Notice "k-NN unscaled": wine's features range from about 0.1 to over 1,000, so the large ones dominate the distance. Scaling is not optional for distance-based models.

Remember: report the mean and spread of CV scores; stratify classification folds; group or time-order folds when rows are not independent.

Hyperparameter tuning

GridSearchCV tries every combination in a grid, scoring each with cross-validation, then refits the best on all the data you gave it. RandomizedSearchCV samples a fixed number of combinations from distributions, which finds good settings far faster when there are many hyperparameters (most of them usually do not matter much).

Remember: grids for one or two knobs, randomized search for many; use step__param names; keep a test set the search never touches.

Class imbalance

Fraud, churn, defects, disease: the interesting class is usually rare. With 5% positives, a model can reach 95% accuracy by never predicting the positive class. Tools that help:

• Better metrics: recall, precision, F1, balanced_accuracy (mean recall across classes), and average_precision (area under the precision-recall curve, which is more informative than ROC-AUC when positives are rare).
• Stratified splits and folds so every fold has some positives.
• Class weights: class_weight="balanced" makes each mistake on a rare sample count more in the loss.
• Threshold tuning: 0.5 is arbitrary. Pick the probability threshold that fits the business cost of false positives vs false negatives, on validation data. (scikit-learn 1.5+ also offers TunedThresholdClassifierCV.)
• Resampling (oversampling the minority, for example with SMOTE from the separate imbalanced-learn package) is sometimes useful, but try weights and thresholds first.

Remember: with rare classes, judge by recall/precision/average precision, stratify, use class weights, and tune the threshold to the costs.

Feature importance and permutation importance

Stakeholders ask "why?". Tree ensembles expose feature_importances_: how much each feature reduced impurity across all splits. It is free but biased: it favors features with many distinct values (an ID column can look important) and is computed on training data.

Permutation importance is model-agnostic and more honest: on held-out data, shuffle one column (breaking its relationship to the target) and measure how much the score drops. A big drop means the model relies on that feature. If two features are strongly correlated, shuffling one barely hurts because the other carries the same information, so read low scores with care.

Remember: importance explains the model, not the world; prefer permutation importance on held-out data; correlated features share credit.

Saving and loading models

Training can take hours; serving must take milliseconds. So we fit once, save the fitted pipeline to a file, and load it in the serving process. scikit-learn's documented options:

• joblib (installed with scikit-learn): joblib.dump(pipe, "model.joblib"), joblib.load("model.joblib"). Efficient for large NumPy arrays. Built on pickle.
• skops: skops.io.dump / skops.io.load use a format that does not execute arbitrary code when loading, and you list which types you trust. Add it with pixi add skops (it is on conda-forge).
• ONNX (via skl2onnx) exports to a portable format served without Python.

Remember: save the whole fitted pipeline; joblib for your own files, skops when files cross trust boundaries; pin versions with pixi.lock.

An end-to-end workflow

Here is the shape of a real project on a tabular dataset: predict whether a subscriber will cancel ("churn") from mixed numeric and categorical data with missing values. We generate the data so the cell works offline, but every step is what you would do with a real CSV loaded by pd.read_csv.

1. Hold out a test set first, stratified.
2. Build a preprocessing ColumnTransformer and a baseline model inside a Pipeline.
3. Compare candidates with cross-validation on the training set.
4. Tune the best candidate.
5. Evaluate once on test; inspect errors and importances.
6. Save the fitted pipeline with its metadata.

Remember: split first, pipeline everything, compare with CV against a dummy baseline, tune, test once, save the pipeline.

Practice

scikit-learn, NumPy, and pandas are available in these exercises.

from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

def build_preprocessor(numeric, categorical):
    num = Pipeline([("impute", SimpleImputer(strategy="median")), ("scale", StandardScaler())])
    cat = Pipeline([
        ("impute", SimpleImputer(strategy="most_frequent")),
        ("onehot", OneHotEncoder(handle_unknown="ignore", sparse_output=False)),
    ])
    return ColumnTransformer([("num", num, list(numeric)), ("cat", cat, list(categorical))])

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

def make_model(C=1.0):
    return Pipeline([("scale", StandardScaler()), ("clf", LogisticRegression(C=C, max_iter=2000))])

def cv_accuracy(model, X, y):
    cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
    return float(cross_val_score(model, X, y, cv=cv).mean())

from sklearn.model_selection import GridSearchCV
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

def tune_knn(X, y):
    pipe = make_pipeline(StandardScaler(), KNeighborsClassifier())
    grid = GridSearchCV(pipe, {"kneighborsclassifier__n_neighbors": [1, 3, 5, 7, 9, 11, 15]}, cv=5)
    grid.fit(X, y)
    return int(grid.best_params_["kneighborsclassifier__n_neighbors"]), float(grid.best_score_)

import numpy as np
from sklearn.inspection import permutation_importance

def top_features(model, X_val, y_val, names, k=3):
    result = permutation_importance(model, X_val, y_val, n_repeats=10, random_state=0)
    order = np.argsort(result.importances_mean)[::-1][:k]
    return [str(names[i]) for i in order]

import numpy as np

def threshold_for_recall(y_true, proba, target_recall):
    y_true, proba = np.asarray(y_true), np.asarray(proba)
    positives = proba[y_true == 1]
    for t in sorted(set(proba.tolist()), reverse=True):
        if np.mean(positives >= t) >= target_recall:
            return t
    return float(proba.min())

Mini-project

Quiz

Recap

• Every estimator has fit; predictors add predict; transformers add transform. Learned attributes end in _.
• Impute, scale, and one-hot encode with a ColumnTransformer; set handle_unknown="ignore".
• Pipelines tie preprocessing to the model so fitting happens only on training folds: no leakage.
• Linear models are fast baselines; random forests are robust; histogram gradient boosting usually wins on tables; scale for k-NN, SVM, and linear models.
• Cross-validate (stratified for classification) and report mean ± std.
• Tune with GridSearchCV or RandomizedSearchCV using step__param names; keep a final test set.
• For imbalance: better metrics, class weights, and threshold tuning.
• Explain with permutation importance on held-out data.
• Persist the whole pipeline; never load untrusted pickles; pin versions with pixi.lock.

Going further

• scikit-learn: getting started
• Pipelines and composite estimators
• Cross-validation: evaluating estimator performance
• Model persistence (joblib, skops, ONNX, and security)
• Permutation feature importance
• Next: M57 · Unsupervised Learning & Embeddings.