M52 · Polars, Arrow & Columnar Data

Row storage vs column storage

A table is two-dimensional, but memory and disk are one long line of bytes. Something has to decide the order. There are two classic choices:

Row-oriented (row store): all fields of record 1, then all fields of record 2, and so on. Databases built for transactions (PostgreSQL, SQLite, MySQL) store rows this way, because their typical work is "fetch or update this one order", and all of an order's fields are next to each other.
Column-oriented (column store): every value of column 1, then every value of column 2. Analytical systems (Arrow, Parquet, Polars, DuckDB, BigQuery, Snowflake) do this, because their typical work is "average the amount column over 50 million rows", which touches one column and every row.

The same table, two byte orders. Analytics usually reads few columns and many rows, so it favors columns.

Column storage has three big payoffs for analytics. Less I/O: a query that uses 3 of 40 columns reads about 3/40 of the data. Faster CPU loops: one column is a dense array of one type, which is exactly what CPU caches and SIMD (single instruction, multiple data) instructions like. Better compression: values of one column resemble each other (many repeated city names, slowly increasing timestamps), so encodings like dictionary and run-length shrink them dramatically.

You already know a column store: a pandas DataFrame keeps each column as its own array, and NumPy arrays are contiguous. Let's measure the effect in pure Python with the array module, which stores raw machine numbers contiguously.

Remember: row stores are for transactions on whole records; column stores are for analytics over a few columns of many rows. Columns give less I/O, faster loops, and better compression.

Apache Arrow: a shared in-memory format

Before Arrow, every tool had its own in-memory layout. Moving a table from a database driver to pandas to a machine-learning library meant serializing (converting to bytes) and deserializing at each step, which often cost more time than the analysis itself. Apache Arrow is a language-independent specification for how columnar data sits in memory. If two libraries both use Arrow, one can hand the other a pointer to the same buffers: zero-copy sharing. Polars, DuckDB, PyArrow, pandas' Arrow-backed dtypes (including the pandas 3.0 str dtype), Spark, and many database drivers speak Arrow.

How an Arrow array is laid out

An Arrow array is a few flat buffers plus metadata (type, length, null count):

A validity bitmap: one bit per value, 1 = present, 0 = null. Nulls cost one bit, and the data buffer keeps a placeholder at that slot so positions stay aligned.
For fixed-width types (int64, float64, timestamps): a data buffer of values packed end to end, exactly like a NumPy array.
For variable-width types (strings, binary): an offsets buffer of n+1 integers and one data buffer holding all the bytes concatenated. String i is data[offsets[i]:offsets[i+1]].

Variable-length strings become offsets plus one byte buffer; nulls live in a separate bitmap.

Arrow groups columns of equal length into a RecordBatch, and a sequence of batches with one schema forms a Table. Batches let engines process large data in chunks and in parallel. The same layout is also used on the wire (Arrow IPC, Arrow Flight) and on disk (the Feather format), so data can move between processes without conversion.

Remember: Arrow is a standard columnar memory layout (validity bitmap, offsets, data buffers). Shared layout means tools can pass tables without copying or converting.

Parquet: columnar files that skip work

Arrow is the in-memory format; Parquet is its on-disk partner. A Parquet file is organized for reading as little as possible:

Rows are split into row groups (often ~100k-1M rows each).
Inside a row group, each column is stored separately as a column chunk, divided into pages.
Pages are encoded (dictionary encoding for repeated values, run-length encoding for runs, delta encoding for sorted numbers) and then compressed (Snappy, Zstandard, gzip, LZ4).
A footer at the end holds the schema and statistics for every column chunk: min, max, and null count.

Readers check the footer first, then fetch only the needed column chunks of row groups that might match.

Two optimizations follow directly from this layout:

Projection pushdown: only the column chunks for the columns a query uses are read. A projection is the list of columns you select.
Predicate pushdown: a predicate is a filter condition. If a row group's statistics prove no row can match (max(amt) <= 100 for amt > 100), the reader skips the whole row group without decompressing it. This works best when data is sorted or clustered by the filtered column.

import random

random.seed(3)
# Sorted-ish data (like a file written in date order): later row groups hold larger values.
values = sorted(random.randint(1, 1000) for _ in range(10_000))
row_groups = [values[i:i + 1000] for i in range(0, len(values), 1000)]
stats = [(min(rg), max(rg)) for rg in row_groups]      # what the footer stores

def scan(lo, hi):
    read = 0
    out = []
    for (mn, mx), rg in zip(stats, row_groups):
        if mx < lo or mn > hi:        # statistics prove nothing in here can match
            continue
        read += 1
        out.extend(v for v in rg if lo <= v <= hi)
    return out, read

matches, groups_read = scan(420, 450)
print(f"matched {len(matches)} rows, read {groups_read} of {len(row_groups)} row groups")

random.shuffle(values)    # the same data, unsorted
row_groups = [values[i:i + 1000] for i in range(0, len(values), 1000)]
stats = [(min(rg), max(rg)) for rg in row_groups]
matches, groups_read = scan(420, 450)
print(f"unsorted: matched {len(matches)} rows, read {groups_read} of {len(row_groups)} row groups")

Encodings you can see

Dictionary encoding replaces repeated values with small integer codes plus a lookup table (exactly what the pandas category dtype does). Run-length encoding (RLE) stores (value, count) pairs, which is tiny for sorted or repetitive columns. Parquet typically dictionary-encodes a column and then run-length encodes the codes.

Remember: Parquet = row groups of column chunks, encoded and compressed, with min/max stats in the footer. Readers skip columns (projection pushdown) and row groups (predicate pushdown). Sorting by common filter columns makes skipping effective.

Polars: DataFrames built from expressions

Polars is a DataFrame library written in Rust, built on the Arrow memory model, multi-threaded by default, and designed around a query optimizer. Its API looks different from pandas in a few deliberate ways:

No row index. Rows are identified by position or by columns you choose. There is no alignment, no loc/iloc distinction, no reset_index.
Expressions. An expression such as pl.col("amt") * 1.25 is a description of a computation, not its result. You pass expressions to contexts: select, with_columns, filter, group_by(...).agg. Because Polars sees the whole description, it can run independent expressions in parallel and optimize them.
Strict types. Each column has one Arrow-based dtype (Int64, Float64, String, Categorical, Datetime, List, Struct); nulls are tracked with validity bitmaps, so integers stay integers when values are missing.
Immutable style. Methods return new DataFrames; there is no in-place assignment.

Task	pandas	Polars
new column	`df.assign(t=df.a * 2)`	`df.with_columns((pl.col("a") * 2).alias("t"))`
filter rows	`df[df.a > 1]`	`df.filter(pl.col("a") > 1)`
pick columns	`df[["a", "b"]]`	`df.select("a", "b")`
group summary	`df.groupby("k").agg(total=("a", "sum"))`	`df.group_by("k").agg(pl.col("a").sum().alias("total"))`
group value per row	`df.groupby("k")["a"].transform("sum")`	`pl.col("a").sum().over("k")`
if/else	`np.where(cond, x, y)`	`pl.when(cond).then(x).otherwise(y)`

To build intuition in the browser, here is the same query in pandas. Notice how each Polars expression maps to a pandas operation, and how Polars needs no lambda to refer to columns created earlier in a chain:

Remember: Polars has no index; you write expressions (pl.col, pl.when, .over) and give them to contexts (select, with_columns, filter, group_by().agg).

The lazy API and the query optimizer

Everything above ran eagerly: each method executed immediately. Polars' lazy API instead builds a query plan (a tree of operations) and runs nothing until you call .collect(). Seeing the whole plan first lets the optimizer rewrite it:

Predicate pushdown: move filters as early as possible, ideally into the file scan so Parquet row groups are skipped.
Projection pushdown: read only columns that some later step uses.
Slice pushdown: head(10) stops reading early when it can.
Common subexpression elimination: compute a repeated expression once.
Expression simplification and choosing join strategies.

Start a lazy query with pl.scan_csv, pl.scan_parquet, or df.lazy(), chain the same methods as before, and finish with .collect() (or .sink_parquet(path) to stream results to a file without holding them all in memory). .explain() prints the optimized plan; .explain(optimized=False) prints the plan as you wrote it. Read plans bottom-up: the scan is at the bottom.

Running that locally prints two plans. The unoptimized plan filters after computing temp_f for every row and reads all three columns. In the optimized plan, the scan line shows SELECTION: col("city") == "Oslo" (the filter moved into the reader) and a projection of 2 of 3 columns (humidity is never read).

To make the idea concrete without Polars, here is a toy lazy frame: it records operations, then a tiny "optimizer" moves filters ahead of computed columns when the filter does not depend on them.

class Lazy:
    def __init__(self, rows, plan=()):
        self.rows, self.plan = rows, list(plan)

def with_column(self, name, fn, uses):
        return Lazy(self.rows, self.plan + [("with_column", name, fn, set(uses))])

def filter(self, fn, uses):
        return Lazy(self.rows, self.plan + [("filter", None, fn, set(uses))])

def optimized(self):
        plan = list(self.plan)
        changed = True
        while changed:                      # bubble filters toward the scan
            changed = False
            for i in range(1, len(plan)):
                prev, cur = plan[i - 1], plan[i]
                if cur[0] == "filter" and prev[0] == "with_column" and prev[1] not in cur[3]:
                    plan[i - 1], plan[i] = cur, prev
                    changed = True
        return plan

def collect(self, optimize=True):
        work = 0
        rows = [dict(r) for r in self.rows]
        for kind, name, fn, _ in (self.optimized() if optimize else self.plan):
            work += len(rows)
            if kind == "with_column":
                for r in rows:
                    r[name] = fn(r)
            else:
                rows = [r for r in rows if fn(r)]
        return rows, work

data = [{"city": c, "temp": t} for c, t in [("Oslo", 3.5), ("Lima", 19.0), ("Oslo", 4.0), ("Pune", 31.0)] * 1000]
q = (
    Lazy(data)
    .with_column("temp_f", lambda r: r["temp"] * 9 / 5 + 32, uses=["temp"])
    .filter(lambda r: r["city"] == "Oslo", uses=["city"])
)
print("plan as written :", [step[0] for step in q.plan])
print("optimized plan  :", [step[0] for step in q.optimized()])
for opt in (False, True):
    rows, work = q.collect(optimize=opt)
    print(f"optimize={opt}: {len(rows)} rows, {work} row-operations")

Remember: lazy = build a plan, optimize, then execute. explain() shows what will actually run; look for filters and column lists pushed into the scan.

Why Polars is fast, and measuring it honestly

Polars is often several times faster than pandas on group-bys, joins, and scans of large files, and uses less memory. The reasons are architectural, not magic:

Parallel by default

Expressions and partitions run on all CPU cores. pandas operations mostly use one core.

Query optimization

Lazy plans skip unneeded columns, rows, and row groups before any work is done.

Arrow memory

Compact strings, bitmaps for nulls, and zero-copy interop with other Arrow tools.

No Python in the loop

Expressions compile to Rust kernels. A Python lambda via map_elements brings back Python speed, so avoid it.

Benchmarks are easy to get wrong. Measure on realistic data sizes (tiny data is dominated by overhead), run several repetitions and take the median or best, include I/O if your real job includes I/O, make sure both tools compute the same answer, and remember pandas 3.0 with Arrow-backed strings narrowed some gaps.

Remember: Polars is fast because of parallelism, optimization, Arrow memory, and keeping Python out of inner loops. Benchmark with realistic sizes, repetitions, and equal answers.

Interoperability: pandas, Polars, PyArrow, DuckDB

Because they share Arrow, these tools convert cheaply, often without copying numeric data:

From	To	Call
Polars	pandas	`pl_df.to_pandas()` (add `use_pyarrow_extension_array=True` to keep Arrow-backed columns)
pandas	Polars	`pl.from_pandas(pd_df)`
Polars	PyArrow	`pl_df.to_arrow()` (zero-copy for most types)
PyArrow	Polars	`pl.from_arrow(table)`
pandas	PyArrow	`pa.Table.from_pandas(pd_df)`; back with `table.to_pandas()`
any of them	DuckDB SQL	`duckdb.sql("SELECT ... FROM pl_df")` reads DataFrames in scope by name

Remember: Arrow is the common language; to_pandas, from_pandas, to_arrow, from_arrow are cheap bridges, and DuckDB can query any of them with SQL.

Choosing pandas, Polars, or DuckDB

All three are excellent; the right one depends on data size, team skills, and the surrounding ecosystem.

	pandas	Polars	DuckDB
Interface	Python methods, row index	Python expressions, no index	SQL (plus a Python relational API)
Execution	eager, mostly single-threaded	eager or lazy, multi-threaded, optimized	lazy, multi-threaded, optimized, out-of-core
Sweet spot	small to medium data, exploration, the widest library ecosystem (statsmodels, scikit-learn, plotting)	medium to large data in Python pipelines; complex transformations	large data, joins and aggregations, querying Parquet/CSV files directly, SQL-fluent teams
Larger than memory	no (chunk manually)	yes, with the streaming engine	yes, spills to disk
Time series and index tricks	strongest	good (`group_by_dynamic`, `rolling`)	good via SQL window functions

A practical rule: start in pandas for small exploratory work and anything that hands a DataFrame to another library that expects pandas; reach for Polars when pandas becomes slow or memory-hungry inside Python code; reach for DuckDB when the natural expression is SQL or the data lives in many Parquet files. They interoperate, so you can mix them in one project. M61 Analytical SQL & DuckDB covers DuckDB in depth.

Remember: choose by data size, interface, and ecosystem; they share Arrow, so mixing is cheap.

Practice

These exercises build the core columnar ideas in plain Python, so they run and grade in the browser.

Write to_columns(rows) that converts a list of dict records into a column store: a dict mapping each column name to a list of values. Columns appear in the order they are first seen. A record missing a key contributes None for that column, so every list has length len(rows).

Hint

First collect column names in order (a dict used as an ordered set works well), then build each list with row.get(name).

Show solution

def to_columns(rows):
    names = {}
    for row in rows:
        for key in row:
            names.setdefault(key, None)
    return {name: [row.get(name) for row in rows] for name in names}

Write dict_encode(values) returning (dictionary, codes): dictionary is the list of distinct non-None values in first-seen order, and codes gives each value's position in the dictionary, with -1 for None (as pandas categoricals do). Also write dict_decode(dictionary, codes) that reverses it.

Hint

Keep a dict from value to code while you scan; append to the dictionary the first time you see a value.

Show solution

def dict_encode(values):
    index = {}
    dictionary, codes = [], []
    for v in values:
        if v is None:
            codes.append(-1)
            continue
        if v not in index:
            index[v] = len(dictionary)
            dictionary.append(v)
        codes.append(index[v])
    return dictionary, codes

def dict_decode(dictionary, codes):
    return [None if c == -1 else dictionary[c] for c in codes]

Write rle_encode(values) that returns a list of (value, run_length) tuples for consecutive equal values, and rle_decode(runs) that expands them back into a list.

Hint

itertools.groupby groups consecutive equal items, or loop and compare with the last run.

Show solution

from itertools import groupby

def rle_encode(values):
    return [(key, sum(1 for _ in group)) for key, group in groupby(values)]

def rle_decode(runs):
    return [value for value, count in runs for _ in range(count)]

A file is a list of row groups; each is a list of numbers. Write build_stats(row_groups) returning a list of (min, max) per group, and scan(row_groups, stats, lo, hi) returning (matches, groups_read): all values with lo <= v <= hi in file order, and how many row groups had to be read. Skip a group without reading it when its stats prove it cannot contain a match.

Hint

A group can be skipped when max < lo or min > hi.

Show solution

def build_stats(row_groups):
    return [(min(rg), max(rg)) for rg in row_groups]

def scan(row_groups, stats, lo, hi):
    matches, read = [], 0
    for rg, (mn, mx) in zip(row_groups, stats):
        if mx < lo or mn > hi:
            continue
        read += 1
        matches.extend(v for v in rg if lo <= v <= hi)
    return matches, read

Build a miniature version of Polars expressions over a column store (a dict of equal-length lists). Implement a class Expr and a function col(name) so that:

col("a").eval(table) returns the list table["a"].
+, -, *, / between two expressions, or an expression and a number (on either side), produce a new Expr that computes element-wise.
>, <, == produce boolean expressions, and & combines two boolean expressions element-wise.
filter_table(table, predicate) returns a new column store keeping rows where the predicate is True.

Nothing is computed until eval is called: an expression is a description.

Hint

Write one helper _binary(self, other, op) that wraps a number in a constant expression, then returns Expr(lambda t: [op(a, b) for a, b in zip(self.eval(t), other.eval(t))]). Each dunder method calls it with an operator function. Defining __eq__ means you should also set __hash__ = None or leave it; tests do not hash expressions.

Show solution

import operator

class Expr:
    def __init__(self, fn):
        self.fn = fn

    def eval(self, table):
        return self.fn(table)

    def _binary(self, other, op, reverse=False):
        if not isinstance(other, Expr):
            value = other
            other = Expr(lambda t: [value] * len(next(iter(t.values()))))
        left, right = (other, self) if reverse else (self, other)
        return Expr(lambda t: [op(a, b) for a, b in zip(left.eval(t), right.eval(t))])

    def __add__(self, other): return self._binary(other, operator.add)
    def __radd__(self, other): return self._binary(other, operator.add, reverse=True)
    def __sub__(self, other): return self._binary(other, operator.sub)
    def __mul__(self, other): return self._binary(other, operator.mul)
    def __rmul__(self, other): return self._binary(other, operator.mul, reverse=True)
    def __truediv__(self, other): return self._binary(other, operator.truediv)
    def __gt__(self, other): return self._binary(other, operator.gt)
    def __lt__(self, other): return self._binary(other, operator.lt)
    def __eq__(self, other): return self._binary(other, operator.eq)
    def __and__(self, other): return self._binary(other, lambda a, b: a and b)
    __hash__ = None

def col(name):
    return Expr(lambda t: list(t[name]))

def filter_table(table, predicate):
    keep = predicate.eval(table)
    return {name: [v for v, k in zip(values, keep) if k] for name, values in table.items()}

Mini-project: port a pandas analysis to Polars lazy

Take an analysis you would write in pandas, port it to the Polars lazy API, prove both give the same answer, and measure the difference on data large enough to matter.

pixi init columnar-bench && cd columnar-bench
pixi add python=3.14 pandas polars pyarrow numpy
pixi run python make_data.py     # writes trips.parquet (10M rows, 20 row groups) and trips.csv
pixi run python bench.py

Data: generate ~10 million synthetic taxi trips with NumPy: pickup_time, zone (260 zones), distance_km, fare, tip, payment_type. Sort by pickup_time and write Parquet with row_group_size=500_000, plus a CSV copy.

Question: for trips in March paid by card, what are the median fare per km and the average tip percentage per zone, for the 10 busiest zones?

A pandas version reads Parquet with columns= and answers the question
A Polars version uses scan_parquet and a single lazy chain ending in collect()
explain() output shows the date filter and column projection pushed into the scan (paste it in a comment)
An assertion proves both versions return the same zones and values (within float tolerance)
Timings use the best of 5 runs, for both Parquet and CSV inputs, printed as a small table
A short README explains which tool you would pick for this job and why

Stretch goals: answer the same question with DuckDB SQL on the Parquet file; try collect(engine="streaming") with 50M rows; compare file sizes for Snappy vs Zstandard compression.

Quiz

Recap

Column stores read less, loop faster, and compress better for analytics.
Arrow standardizes columnar memory: validity bitmaps, offsets, data buffers, record batches; zero-copy interop.
Parquet stores row groups of encoded, compressed column chunks with min/max stats for skipping.
Polars: no index, expressions in contexts, strict Arrow types, parallel execution.
Lazy plans let the optimizer push down predicates and projections; check with explain().
pandas, Polars, PyArrow, and DuckDB convert cheaply; choose by size, interface, and ecosystem.

Going further

Polars user guide (expressions, lazy API, and "Coming from pandas")
Arrow columnar format specification
Parquet file format documentation
PyArrow documentation
Next: M53 Data Visualization; later, M61 Analytical SQL & DuckDB