Parquet

omniload reads Apache Parquet files. Parquet is a column-oriented binary format for structured data. It stores a schema with the data and supports nested values, typed columns, compression, and row groups.

Parquet is available for read operations on every supported filesystem source. Parquet is also available for write operations through the local file:// destination.

Installation

Parquet support is included in the base omniload installation. It uses pyarrow to read Parquet files and polars to write them.

pip install omniload

Do not install an optional extra to use Parquet.

Where it works

Every source that uses the shared filesystem readers can read Parquet:

The source determines the storage connection and authentication. Parquet adds no storage-specific configuration.

omniload selects the Parquet reader in one of these cases:

  • The filename ends in .parquet.

  • The filename ends in .parquet.gz.

  • The source path has an explicit #parquet format hint.

For example, use #parquet when the file has no .parquet extension:

file://data/events.bin#parquet

Gzipped Parquet files are decompressed before pyarrow reads them:

s3://my-bucket/events/2026-09-01.parquet.gz

See Filesystem for format detection, glob patterns, compression, and incremental file selection.

Reading behavior

omniload reads Parquet files with pyarrow.parquet.ParquetFile. The reader returns rows in batches. The default batch size is 10 rows.

The reader converts each Arrow batch to Python dictionaries before it passes the rows to the loader. The Parquet schema controls the decoded column types.

Parquet is not a streaming format over a remote transport. pyarrow reads the file footer and can request the complete data section. A large remote Parquet file can therefore require substantial memory and network transfer even when the reader returns rows in batches.

If the Parquet file is corrupt, truncated, encrypted without the required configuration, or uses an unsupported codec, pyarrow raises an error during the load. Validate files upstream when a partial or failed load is not acceptable.

Examples

Load a local Parquet file into DuckDB

omniload ingest \
    --source-uri 'file://data/events.parquet' \
    --source-table 'events' \
    --dest-uri 'duckdb:///local.duckdb' \
    --dest-table 'public.events'

When the URI already contains the file path, --source-table does not select a table inside the Parquet file. The destination table is set by --dest-table.

Load a Parquet file from S3 into DuckDB

omniload ingest \
    --source-uri 's3://my-bucket?access_key_id=YOUR_ACCESS_KEY&secret_access_key=YOUR_SECRET_KEY' \
    --source-table 'events/2026-09-01.parquet' \
    --dest-uri 'duckdb:///local.duckdb' \
    --dest-table 'public.events'

Use the documentation for the selected filesystem source to configure authentication and source paths.

Read a file with a non-standard extension

omniload ingest \
    --source-uri 'file://data/events.data#parquet' \
    --dest-uri 'duckdb:///local.duckdb' \
    --dest-table 'public.events'

The #parquet fragment is not part of the filename. It instructs omniload to use the Parquet reader.

Write a local Parquet file

Use a file:// destination path that ends in .parquet, or use #parquet:

omniload ingest \
    --source-uri 'postgres://user:password@host:5432/app' \
    --source-table 'public.events' \
    --dest-uri 'file://export/events.parquet' \
    --dest-table 'public.events'

The file:// destination writes one Parquet file at the requested path. It creates missing parent directories and overwrites an existing output file.

The destination removes dlt bookkeeping columns before it writes the file. It collects all loaded rows before it writes the Parquet table. This makes a single-file output reliable, but it is not suitable for data that cannot fit in memory.

The column types in that file are decided by the staging format rather than by Parquet: under the default staging a timestamp and a decimal both arrive as text, so they are written as string columns. Pass --loader-file-format parquet for typed columns. See What a load delivers.

See Destination connector for the complete URI and destination-table rules for the file:// destination.

Extended-type handling

This section describes the reader and the writer called directly. An ingest stages rows between the two, and what a load delivers to a file is decided there rather than here: see What a load delivers.

Read directly, strings, integers, floating-point values, booleans, dates, timestamps, times, binary values, decimals, lists, structs and all-null columns all come back as themselves. A time zone survives the read: a timestamp[us, tz=UTC] column arrives as a timezone-aware datetime, and a naive one stays naive.

Nanosecond columns are the exception, because a row carries Python values rather than Arrow ones. A time64[ns] narrows at the read, datetime.time having no nanoseconds, so 00:00:00.123456789 arrives as 00:00:00.123456; passed back to the writer it becomes a time64[ns] column again, carrying the narrowed value. Nanosecond timestamps and durations keep their precision at the read, the pandas types carrying it, and lose it on the way back out, where the writer emits microsecond columns. The Feather reader answers the same way, which the test suite pins; the file itself stores whatever precision it was written with.

The writer has limits of its own, and they follow from the library split: the reader is pyarrow, the writer is polars. Polars widens an integer above the signed 64-bit range to a 128-bit one, and Parquet has no type for that, so the writer narrows such a column to an unsigned 64-bit integer before writing it, inside a struct or a list of structs as well as at the top level. A value in 0..2**64-1 is written as uint64 and reads back digit for digit; anything else, a value past 2**64-1 or a negative in a column another row widened, is refused by value and by the column holding it, and no file is written. Inside a bare list the value is named and the column is not, which is Polars’ own error reporting. write_feather and write_orc build a PyArrow table from the same Python values, where inference stops at a signed 64-bit, and raise OverflowError on the whole range instead. The JSON, JSONL, CSV and YAML writers keep any of these values digit for digit.

A decimal too wide for a 128-bit store goes the other way, and it is the value that decides rather than the column’s declared type: a row carries a Python Decimal, and every writer infers from that one. A decimal256(41, 2) column holding 3.14 reads and writes without complaint, as decimal128(38, 2). One holding a 41-digit value reads back exactly and then fails the Parquet write with Decimal is too large to fit in Decimal128, where write_feather takes it and write_orc refuses it too.

Parquet files and Parquet loader files

The Parquet source format is independent of the Parquet loader format that omniload can select for some warehouse destinations. A Parquet source controls how omniload reads input files. A Parquet loader controls how omniload stages rows for a destination.

You do not need to set a loader option to read a Parquet source file.