Statistics & Probability

Describing data: center, spread, position

Descriptive statistics summarize a dataset you have. (Inferential statistics, later, reasons from a sample to a larger population.) Three families of numbers do most of the work:

• Center: the mean (sum divided by count), the median (the middle value when sorted), and the mode (the most common value). The mean uses every value, so a single extreme value can drag it far; the median is robust to outliers.
• Spread: the variance is the average squared distance from the mean; the standard deviation (std) is its square root, back in the data's units. The interquartile range (IQR) is the distance between the 25th and 75th percentiles: the spread of the middle half, robust like the median.
• Position: quantiles (percentiles) say what fraction of data falls below a value. The 90th percentile response time is the value 90% of requests beat.

Population vs sample variance

If your data is the entire population, variance divides by n. If it is a sample used to estimate a population's variance, divide by n - 1 (Bessel's correction): the sample mean is closer to the sample's own points than the true mean is, so dividing by n underestimates the spread. Libraries disagree on the default, which is a classic bug:

Remember: mean and std are sensitive to outliers; median and IQR are robust. Know whether your tool divides by n or n - 1.

Correlation is not causation

The Pearson correlation coefficient r measures how well two variables fit a straight line, from -1 (perfect negative) through 0 (no linear relationship) to +1 (perfect positive). Spearman's correlation is Pearson's applied to ranks, so it captures any monotonic (always increasing or always decreasing) relationship and resists outliers.

Correlation can arise without one variable causing the other: a confounder (a third variable) can drive both, the causation can run backwards, or the pattern can be coincidence in a small sample. Ice-cream sales and drownings correlate because hot weather drives both. Only a controlled experiment (or careful causal methods) supports a causal claim, which is why A/B tests matter.

Remember: correlation measures association, not cause. Look for confounders, reverse causation, and chance; use experiments when you need causal answers.

Probability by simulation

A probability is the long-run fraction of times an outcome occurs. That definition suggests a method: simulate the process many times and count. This is the Monte Carlo method, and with a computer it answers questions whose formulas are hard.

Python offers two random number sources. The stdlib random module is fine for small simulations. NumPy's Generator, created with np.random.default_rng(seed), draws whole arrays at once (much faster) and is the modern recommended API. A seed makes the pseudo-random sequence reproducible: same seed, same numbers, which is essential for tests and for others re-running your analysis.

Monte Carlo estimate of π

Throw random points into the unit square. The fraction landing inside the quarter circle of radius 1 approaches its area, π/4. Error shrinks like 1/√n: 100 times more points buy one more correct digit.

The birthday paradox

How many people must be in a room before two share a birthday with probability over 50%? Intuition says around 183; the answer is 23, because what matters is the number of pairs (23 people form 253 pairs). The exact probability that n people all have different birthdays is (365/365)(364/365)...((365-n+1)/365).

Remember: probability = long-run frequency; simulate, count, and compare with exact answers when you can. Seed everything.

Probability distributions with scipy.stats

A random variable is a quantity whose value comes from a random process. Its distribution says how likely each value is. Discrete variables (counts) have a probability mass function (PMF): P(X = k). Continuous variables (times, heights) have a probability density function (PDF): probabilities are areas under the curve, and P(X = exactly 1.7) is 0. Both have a cumulative distribution function (CDF): P(X ≤ x).

Every scipy.stats distribution object shares the same methods:

• pmf(k) (discrete) / pdf(x) (continuous): probability mass or density.
• cdf(x): P(X ≤ x). sf(x) = 1 - cdf(x), the survival function (more precise in the tail).
• ppf(q): the percent point function, the inverse of the CDF: "which x has q of the probability below it?" norm.ppf(0.975) is 1.96.
• rvs(size=..., random_state=rng): random variates (samples).
• mean(), std(), interval(0.95).

Remember: PMF/PDF for "how likely", CDF for "at most", sf for "more than", ppf for "which value has this percentile", rvs to simulate.

The law of large numbers and the central limit theorem

Two theorems justify almost everything in inferential statistics.

The law of large numbers (LLN): as the number of independent observations grows, their average converges to the true expected value. It is why casinos always win in the long run and why your Monte Carlo estimates improve with more samples. It does not say short runs "balance out": after five heads, a fair coin is still 50/50 (believing otherwise is the gambler's fallacy).

The central limit theorem (CLT): the average of n independent draws from almost any distribution with finite variance is approximately normally distributed when n is large enough, with mean μ and standard deviation σ/√n. That σ/√n is the standard error (SE) of the mean: the typical distance between a sample mean and the true mean. The CLT is why normal-based confidence intervals and tests work even when the data itself is skewed.

Remember: averages converge (LLN) and are approximately normal with standard error σ/√n (CLT). To halve the error, you need four times the data.

Sampling and bias

A population is everyone (or everything) you want to learn about; a sample is the part you actually measure. A statistic computed on the sample (sample mean x̄) estimates a parameter of the population (true mean μ). Two kinds of error separate them:

• Sampling error: random variation from which units happened to be chosen. It shrinks with larger samples (that is the σ/√n), and statistics quantifies it.
• Bias: systematic error from how the sample was chosen or measured. It does not shrink with more data; a biased sample of a million is still wrong, just precisely wrong.

Common biases: selection bias (an online survey misses people who are offline), survivorship bias (studying only companies that still exist), non-response bias (unhappy customers skip the survey), convenience sampling (asking whoever is nearby), and measurement bias (leading questions, miscalibrated sensors). The defense is a random sample (every unit has a known, nonzero chance of selection) or, for causal questions, random assignment to treatment and control.

Remember: more data shrinks random error, never bias. Fix bias with random sampling or random assignment, and describe exactly which population a sample represents.

Confidence intervals: analytic and bootstrap

A point estimate ("mean delivery time 31.4 minutes") hides uncertainty. A confidence interval (CI) gives a range built by a procedure that captures the true parameter in a stated fraction of repeated samples. For a mean with unknown population σ, the classic 95% interval is

x̄ ± t* × s / √n      # t* = stats.t.ppf(0.975, df=n - 1), s = sample std (ddof=1)

The t distribution replaces the normal because s is itself estimated; it has fatter tails for small n and approaches the normal (t* → 1.96) as n grows.

Interpretation, carefully: "95% confidence" is a property of the method. If you repeated the study many times, about 95% of the intervals you computed would contain the true mean. For any one computed interval, the true mean is either in it or not; it is not correct to say "there is a 95% probability μ is in [29.8, 33.0]" in the frequentist framework.

The bootstrap

What about a CI for a median, a 90th percentile, or a ratio, where no simple formula exists? The bootstrap (Efron, 1979) treats the sample as a stand-in for the population: resample it with replacement many times (each resample the same size as the original), compute the statistic on each, and take the middle 95% of those values (the percentile method). It needs only computation and the assumption that the sample is representative.

Remember: a CI expresses the precision of an estimate; 95% describes the long-run hit rate of the method. Use t-intervals for means and the bootstrap for anything else.

Hypothesis testing and p-values

A hypothesis test asks whether data are surprising under a default assumption. The vocabulary:

• Null hypothesis H₀: the "nothing is going on" claim, such as "the new checkout converts at the same rate as the old one".
• Alternative H₁: what you suspect instead ("the rates differ"). Two-sided alternatives allow a difference in either direction; one-sided alternatives must be chosen before seeing data.
• Test statistic: a number measuring how far the data are from what H₀ predicts (a t value, a chi-square value, a difference in means).
• p-value: the probability, computed assuming H₀ is true, of getting a test statistic at least as extreme as the one observed.
• Significance level α (often 0.05), fixed in advance: reject H₀ if p < α.

Two kinds of mistakes are possible. A Type I error (false positive) rejects a true H₀; its rate is α. A Type II error (false negative) misses a real effect; its rate is β, and power = 1 - β is the probability of detecting an effect of a given size. Power grows with sample size and effect size, so plan the sample size before the study.

A permutation test: the logic without formulas

If H₀ says the group labels do not matter, then shuffling the labels should produce differences as large as the observed one fairly often. Count how often: that fraction is a p-value.

t-tests and chi-square tests in scipy

Remember: a p-value is P(data at least this extreme | H₀), nothing more. Fix α and the alternative in advance; pair tests with confidence intervals and effect sizes.

A/B testing and the multiple-comparison trap

An A/B test is a randomized controlled experiment: users are randomly assigned to version A (control) or B (treatment), and a metric decided beforehand (conversion rate, revenue per visitor) is compared. Randomization balances confounders on average, so a difference can be attributed to the change. The workflow:

1. Choose one primary metric and the minimum detectable effect worth acting on.
2. Compute the sample size for the desired power (commonly 80%) at your α.
3. Randomize, run until the planned sample size is reached (do not stop early when it looks good).
4. Test, report the effect with a CI, and check guardrail metrics (errors, latency, refunds).

Reading the result: the new checkout converted 1.5 percentage points better; if the versions truly converted equally, a difference this large would occur in under 3% of experiments (p ≈ 0.027); and the plausible range for the true lift is about +0.2 to +2.8 points. Whether to ship depends on that range versus the cost of the change, not on p alone.

Multiple comparisons

Each test at α = 0.05 has a 5% false-positive rate when nothing is going on. Run 20 independent tests (20 metrics, 20 segments, or checking results every day) and the chance of at least one false positive is 1 - 0.95²⁰ ≈ 64%. This is how "the new button increases revenue for left-handed users on Tuesdays" gets discovered. Defenses:

• Pre-register one primary metric and a fixed sample size; label everything else exploratory.
• Bonferroni: test each of m hypotheses at α/m (simple, conservative). Holm: a step-down version with more power. Benjamini-Hochberg controls the false discovery rate when you test many hypotheses.
• Do not peek and stop at the first p < 0.05; use a fixed horizon or a sequential method designed for it.

Remember: randomize, fix the metric and sample size in advance, report the difference with a CI, and correct for multiple comparisons (or avoid them).

Effect sizes: how big, not just whether

Statistical significance answers "is there evidence of some difference?" An effect size answers "how big is it?" Always report both. Common effect sizes:

• Raw difference in meaningful units (minutes saved, percentage points of conversion), with a CI. Usually the most useful for decisions.
• Relative lift: difference divided by the baseline (+15.8% conversion).
• Cohen's d: the difference in means divided by the pooled standard deviation, a unitless size. Rough conventions: 0.2 small, 0.5 medium, 0.8 large (context matters more than the labels).

Remember: report the effect size with its CI in real units first; the p-value is supporting evidence, not the headline.

Practice

These exercises use the standard library, NumPy, and SciPy; they run in the browser. Use the seeds given so the results are reproducible.

import statistics

def summarize(values):
    q1, _, q3 = statistics.quantiles(values, n=4)
    return {
        "mean": statistics.mean(values),
        "median": statistics.median(values),
        "stdev": statistics.stdev(values),
        "iqr": q3 - q1,
        "range": max(values) - min(values),
    }

import numpy as np

def estimate_pi(n, seed):
    rng = np.random.default_rng(seed)
    x, y = rng.random(n), rng.random(n)
    return 4 * float(np.mean(x * x + y * y <= 1))

import numpy as np

def p_shared(n, days=365):
    p_distinct = 1.0
    for k in range(n):
        p_distinct *= max(days - k, 0) / days
    return 1 - p_distinct

def simulate_shared(n, trials, seed, days=365):
    rng = np.random.default_rng(seed)
    birthdays = np.sort(rng.integers(0, days, size=(trials, n)), axis=1)
    return float((np.diff(birthdays, axis=1) == 0).any(axis=1).mean())

import numpy as np

def bootstrap_ci(data, stat, n_boot=2000, confidence=0.95, seed=0):
    rng = np.random.default_rng(seed)
    data = np.asarray(data)
    values = [stat(rng.choice(data, size=data.size, replace=True)) for _ in range(n_boot)]
    tail = (1 - confidence) / 2 * 100
    low, high = np.percentile(values, [tail, 100 - tail])
    return float(low), float(high)

import math
from scipy import stats

def ab_test(conv_a, n_a, conv_b, n_b):
    p_a, p_b = conv_a / n_a, conv_b / n_b
    pooled = (conv_a + conv_b) / (n_a + n_b)
    se = math.sqrt(pooled * (1 - pooled) * (1 / n_a + 1 / n_b))
    z = (p_b - p_a) / se
    return {"diff": p_b - p_a, "z": z, "p_value": float(2 * stats.norm.sf(abs(z)))}

import statistics

def cohens_d(a, b):
    na, nb = len(a), len(b)
    pooled = ((na - 1) * statistics.variance(a) + (nb - 1) * statistics.variance(b)) / (na + nb - 2)
    return (statistics.mean(b) - statistics.mean(a)) / pooled ** 0.5

Mini-project: analyze a simulated A/B test and recommend

pixi init ab-analysis && cd ab-analysis
pixi add python=3.14 numpy scipy pandas matplotlib
pixi run python simulate.py      # writes visitors.csv (seeded)
pixi run python analyze.py       # prints results, saves figures

Quiz

Recap

• Center, spread, position: median and IQR are robust; know ddof.
• Correlation is association; confounders and reverse causation are everywhere.
• Simulate to estimate probabilities; seed your generators.
• scipy.stats: pmf/pdf, cdf/sf, ppf, rvs for normal, binomial, Poisson, exponential, uniform.
• LLN: averages converge. CLT: averages are approximately normal with SE σ/√n.
• More data reduces random error, not bias; randomize.
• CIs: t-interval for means, bootstrap for anything; 95% is the method's hit rate.
• p-value = P(data at least this extreme | H₀). Fix α and hypotheses first; correct for multiple tests; report effect sizes.

Going further

• statistics module documentation
• scipy.stats reference and NumPy random Generator
• The ASA statement on p-values
• Think Stats by Allen Downey (free online, Python-based)
• OpenIntro Statistics (free textbook)
• Next: Project 7: City Data EDA Report, then M55 Machine Learning Foundations