Import Parquet Files to PostgreSQL

Fast, parallel file import using DuckDBStream

Source

Parquet

.parquet files

FastTransfer

DuckDBStream

Target

PostgreSQL

Terminal

FastTransfer.exe `
  --sourceconnectiontype "duckdbstream" `
  --sourceserver ":memory:" `
  --query "SELECT * FROM read_parquet('D:\path\to\files\*.*.parquet', filename=true)" `
  --targetconnectiontype "postgresql" `
  --targetserver "your-server" `
  --targetdatabase "your-database" `
  --targettable "your-table" `
  --method "DataDriven" `
  --distributekeycolumn "filename"  `
  --datadrivenquery "select file from glob('D:\path\to\files\*.parquet')"  `
  --degree -2  `
  --loadmode "Truncate"  `
  --mapmethod "Name"

Related Pipelines

✨ Key Features

•Parallel file processing with DuckDBStream
•Streaming architecture - minimal memory usage
•Wildcard pattern support for multiple files
•Automatic schema detection

🚀 Performance

•DataDriven parallelization
•Configurable degree of parallelism
•Optimized bulk loading
•Production-ready reliability