Convert CSV to Parquet with Python and PyArrow
Run this on your own computer using Python 3. Save a UTF-8, comma-delimited input.csv in the same folder as the script. The first record must contain unique, nonempty headers; quoted commas and line breaks are parsed as fields.
Before you start
- Python 3 installed. Package commands below use a local virtual environment on macOS/Linux; on Windows use .venv\Scripts\python.exe in place of .venv/bin/python.
- Keep the original CSV. These examples read the complete table into memory and create separate output files.
- Save the Python code as convert_csv.py beside input.csv.
- Install the listed package when required, then run the script. Check the generated output before using it elsewhere.
python3 -m venv .venv
.venv/bin/python -m pip install pyarrowimport csv
from pathlib import Path
with open("input.csv", encoding="utf-8-sig", newline="") as source:
reader = csv.reader(source, strict=True)
headers = next(reader)
rows = list(reader)
if not headers or any(not h for h in headers) or len(set(headers)) != len(headers):
raise ValueError("Use unique, nonempty column headers")
if any(len(row) != len(headers) for row in rows):
raise ValueError("Every record must have the same number of fields as the header")
import pyarrow as pa
import pyarrow.parquet as pq
columns = [pa.array([row[i] for row in rows], type=pa.string()) for i in range(len(headers))]
table = pa.Table.from_arrays(columns, names=headers)
pq.write_table(table, "output.parquet", compression="snappy").venv/bin/python convert_csv.pyLimits and output
- Writes one STRING column per header, preserving leading zeros and empty strings without numeric/date inference. This example rejects blank or duplicate headers; the site uses stable display names. It loads the parsed CSV into memory.
Documentation: Python CSV reader and writer; Apache Arrow Parquet writing