Vortex

omniload reads the highly performant Vortex file format. Vortex is an extensible columnar data format considered faster than Parquet.

Vortex is supported for reads on shared filesystem sources and for writes through the local file:// destination.

Installation

Vortex support ships in the optional vortex extra, which needs Python 3.11 or newer, so it is not part of the base install:

pip install 'omniload[vortex]'

The full extra includes it. If a Vortex file is read or written without the extra installed, omniload fails with an error naming the extra to install.

Where it works

Vortex is available on every source that uses the shared file readers:

Remote reads use the source’s existing fsspec handle. They use its existing authentication. No separate Vortex storage configuration is required.

A file is read as Vortex when its extension is .vortex (optionally .vortex.gz). You can also append the #vortex format hint to a file with a different extension. Gzipped files are decompressed automatically.

The Vortex library opens files by path only, so a remote or gzipped file is first copied to a local temporary file, which needs enough free disk space to hold it. The file is then scanned in batches, and #chunksize= sets how many rows are handed downstream at a time.

For details about format selection, see File format routing.

Examples

Load a local Vortex file into DuckDB

omniload ingest \
    --source-uri 'file://events/day.vortex' \
    --source-table 'events' \
    --dest-uri duckdb:///local.duckdb \
    --dest-table 'public.events'

Load a Vortex file from S3

Use #vortex if the object name does not end in .vortex.

omniload ingest \
    --source-uri 's3://' \
    --source-table 'my_bucket/events/day.data#vortex' \
    --dest-uri duckdb:///local.duckdb \
    --dest-table 'public.events'

Load multiple Vortex files

Use a glob to load rows from all matching Vortex files.

omniload ingest \
    --source-uri 'file://events/*.vortex' \
    --source-table 'events' \
    --dest-uri duckdb:///local.duckdb \
    --dest-table 'public.events'

Write a source table to a local Vortex file

omniload ingest \
    --source-uri 'postgres://user:password@host:5432/db' \
    --source-table 'public.events' \
    --dest-uri 'file://export/events.vortex' \
    --dest-table 'public.events'

Vortex output is available through the local file:// destination. Columns that are absent from an individual source row are written as null values. A timestamp keeps its zone when that is a named IANA zone the Vortex library carries. Any other, such as a fixed +12:00 offset, is written as the same instant in UTC.

Note

dlt stages the rows between the source and the writer, and the staging format decides what the writer receives: a load to a local file:// destination writes a timestamp or a decimal as text unless --loader-file-format parquet is passed. See What a load delivers.