Files & Data Formats

Paths with pathlib

A path names a location in the file system, such as /home/ada/reports/jan.csv or C:\Users\ada\reports\jan.csv. Paths are not just strings: they have parts (folders, a file name, an extension), they differ between operating systems, and joining them by hand with + "/" + is a classic source of bugs. The pathlib module represents a path as a Path object with methods for everything you need.

A relative path such as data/jan.csv is interpreted relative to the current working directory, the folder your program was started from, which is not necessarily the folder containing your script. When a script needs files next to itself, build the path from Path(__file__).parent.

Asking the file system

The browser runtime has a real (in-memory) file system, so the following code creates folders and files.

Python 3.14 adds two long-requested methods. Path.copy(target) copies a file or a whole directory tree and returns the new path; Path.move(target) moves (renames) it, even across file systems. Before 3.14 you needed shutil.copy2, shutil.copytree, and shutil.move.

Opening files: modes, text, and bytes

Path.read_text() and write_text() are perfect for small files. For anything more, open a file object with open() (or path.open()) and a mode:

Text mode (the default) decodes bytes to str using an encoding, and translates line endings so you always see "\n". Binary mode gives you the raw bytes, untouched. Use binary for images, archives, compressed data, and anything that is not text, and for formats whose library insists on it (tomllib).

Eleven characters of text became thirteen bytes, because é and ï each take two bytes in UTF-8. Text is characters; files store bytes; an encoding is the rule between them.

Encodings

Recall from M03 that a str is a sequence of Unicode code points and bytes are raw numbers from 0 to 255. UTF-8 is the universal encoding today: it handles every language and emoji, and plain ASCII text is unchanged. Other encodings still appear in old files: latin-1, cp1252 (old Windows), utf-16.

If you do not pass encoding=, Python 3.14 uses the locale's preferred encoding, which is UTF-8 on macOS and Linux but may be something else on Windows. The same program can then read a file correctly on one machine and garble it on another. Always pass encoding="utf-8" for text files.

The errors= argument decides what happens with undecodable bytes: "strict" (the default, raise), "replace", "ignore", or "backslashreplace". Prefer "strict" so problems are noticed. Some programs, notably Excel, write a byte-order mark (BOM) at the start of UTF-8 files; the "utf-8-sig" encoding removes it when reading.

The with statement and context managers

An open file holds an operating-system resource. If you forget to close() it, written data may never reach the disk, and long-running programs can run out of file handles. Calling close() yourself is fragile: an exception before that line skips it.

The with statement solves this. The object after with is a context manager: it sets something up on entry and is guaranteed to clean up on exit, whether the block finishes normally, returns, or raises. It is the try/finally from M07, packaged.

f = open(path, encoding="utf-8")
try:
    text = f.read()
finally:
    f.close()

with open(path, encoding="utf-8") as f:
    text = f.read()
# f is closed here, even after an exception

Step through and watch closed change when the block ends, even though the second block raises:

One with can manage several resources, and you can write your own context managers with contextlib.contextmanager (the full story is in M15):

Reading large files line by line

f.read() loads the whole file into memory. That is fine for kilobytes, not for a 20 GB log file. A text file object is an iterator over lines: a for loop reads one line at a time, so memory use stays tiny no matter how big the file is.

• Each line includes its trailing "\n" (except possibly the last); use .rstrip("\n").
• f.readline() reads one line; f.readlines() reads all lines into a list (avoid for big files).
• For binary data, read fixed-size chunks: while chunk := f.read(65536): ....

CSV files

CSV (comma-separated values) is the lingua franca of spreadsheets and bank exports. It looks trivial, but values can contain commas, quotes, and even newlines, so never split lines on "," yourself. Use the csv module, and always open CSV files with newline="" so the module can handle line endings itself (the docs require it).

• csv.reader/csv.writer work with lists; DictReader/DictWriter use the header row as dict keys, which is clearer and survives column reordering.
• Every value read is a string. Convert numbers and dates yourself, and validate them.
• Other dialects: delimiter=";" (common in Europe) or delimiter="\t" for TSV.
• reader.line_num gives the current line number for error messages.

The pipeline visualizer shows the shape of almost every data job, including this module's mini-project and Project 1: read, parse, transform, write.

JSON

JSON (JavaScript Object Notation) is the standard format for web APIs and structured data. It maps neatly onto Python:

Four functions: dumps (to string) and loads (from string), dump (to a file) and load (from a file). The "s" means string.

Custom types: default=

JSON has no dates, decimals, sets, or paths. json.dumps raises TypeError for them unless you pass default=, a function called for each object JSON cannot encode. It returns something JSON can encode, or raises TypeError itself.

TOML configuration with tomllib

TOML is a configuration format designed to be easy for humans to read and write. You have already seen it: pixi.toml and pyproject.toml. The standard library's tomllib reads TOML (writing needs a third-party package). It requires the file to be opened in binary mode, because TOML is defined to be UTF-8 and the parser decodes it itself.

Compression: gzip and compression.zstd

Text data compresses extremely well. The gzip module reads and writes .gz files with the same open() interface as normal files, so streaming code barely changes. Use "wt"/"rt" for text and "wb"/"rb" for bytes.

Temporary files and shutil

Tests and data jobs often need scratch space that cleans itself up. tempfile.TemporaryDirectory() is a context manager that creates a fresh directory and deletes it, with everything inside, on exit. NamedTemporaryFile does the same for a single file. (pytest gives tests a tmp_path fixture built on the same idea; Project 1's tests use it.)

shutil ("shell utilities") handles whole-file and whole-tree operations: copy2 (copy with metadata), copytree, move, rmtree (delete a tree, carefully!), make_archive/unpack_archive, and disk_usage.

Practice

These exercises write to temporary directories; the tests create the input files for you.

from pathlib import Path


def count_file(path):
    lines = words = chars = 0
    with open(path, encoding="utf-8") as f:
        for line in f:
            lines += 1
            words += len(line.split())
            chars += len(line)
    return lines, words, chars

import csv


def load_scores(path):
    scores = {}
    with open(path, encoding="utf-8", newline="") as f:
        reader = csv.DictReader(f)
        for row in reader:
            try:
                scores[row["name"].strip()] = int(row["score"])
            except ValueError as e:
                raise ValueError(f"line {reader.line_num}: bad score {row['score']!r}") from e
    return scores

import csv


def save_rows(path, rows):
    with open(path, "w", encoding="utf-8", newline="") as f:
        writer = csv.DictWriter(f, fieldnames=list(rows[0]))
        writer.writeheader()
        writer.writerows(rows)
    return len(rows)

import json
from datetime import date
from decimal import Decimal


def _encode(obj):
    if isinstance(obj, date):
        return obj.isoformat()
    if isinstance(obj, Decimal):
        return str(obj)
    if isinstance(obj, set):
        return sorted(obj)
    raise TypeError(f"cannot encode {type(obj).__name__}")


def to_json(data):
    return json.dumps(data, indent=2, sort_keys=True, default=_encode)

import tomllib

DEFAULTS = {"host": "127.0.0.1", "port": 8000, "debug": False}


def read_server_config(path):
    with open(path, "rb") as f:
        data = tomllib.load(f)
    return DEFAULTS | data.get("server", {})

import gzip
import shutil
from pathlib import Path


def backup_text_files(src, dest):
    written = []
    for path in src.rglob("*.txt"):
        rel = path.relative_to(src)
        target = dest / rel.with_name(rel.name + ".gz")
        target.parent.mkdir(parents=True, exist_ok=True)
        with path.open("rb") as fin, gzip.open(target, "wb") as fout:
            shutil.copyfileobj(fin, fout)
        written.append(target.relative_to(dest).as_posix())
    return sorted(written)

Mini-project: a CSV → JSON report converter

mkdir csv2report && cd csv2report
pixi init
pixi add python=3.14
pixi run python csv2report.py data/*.csv --out report.json
pixi run python csv2report.py data/*.csv --out report.json.gz    # compressed output

{
  "files": ["data/jan.csv", "data/feb.csv"],
  "rows": 1840,
  "revenue_by_region": {"EU": "18250.40", "US": "22711.95"},
  "units_by_product": {"pen": 1200, "notebook": 640},
  "best_seller": "pen",
  "rejected": [{"file": "data/feb.csv", "line": 17, "error": "qty must be an integer, got 'ten'"}]
}

Quiz

Recap

• pathlib.Path for every path: / to join, properties to inspect, glob/rglob to search, 3.14 copy/move.
• Modes: r, w (erases), a, x; add b for bytes.
• Always pass encoding="utf-8"; utf-8-sig for BOMs; UTF-8 becomes the default in 3.15.
• with guarantees cleanup; iterate files to stream lines.
• CSV: newline="", DictReader/DictWriter, values are strings.
• JSON: dump/load/dumps/loads, default= for custom types.
• TOML: tomllib.load(open(p, "rb")), read-only.
• gzip.open and 3.14's compression.zstd.open work like open; tempfile and shutil handle scratch space and trees.

Going further

• pathlib: object-oriented file-system paths
• Python tutorial: Reading and writing files
• Unicode HOWTO
• csv, json, and tomllib
• compression.zstd and PEP 784
• tempfile and shutil
• Next: Project 1: Budget Ledger CLI