Parquet Formatting

Apache Parquet is a columnar storage format optimized for analytics. FastTransfer automatically uses Parquet format when the output file has a .parquet extension.

Parquet Compression

Use the --parquetcompression parameter to specify the parquet compression codec.

./FastTransfer \
...
 --fileoutput "orders.parquet" \
 --parquetcompression Snappy \
...

Syntax:

• Long form: --parquetcompression <algorithm>

Default: Zstd (faster and better compression ratio than others)

Available Algorithms:

• None - No compression
• Snappy - Fast, small compression, classic
• Gzip - Smaller files, slower
• Lzo - Slower, medium compression (rarely used)
• Lz4 - Slower, moderate compression
• Zstd - Best compression and Fast (default)

tip

The Zstd (default) is best for most cases. It provides the best compression ratio with moderate speed, ideal for cloud storage to minimize storage costs and transfer time.

• So no need to set this parameter to use Zstd.

Complete Example

Windows

.\\FastTransfer.exe `
  --connectiontype mssql `
  --server "localhost" `
  --database "sales" `
  --trusted `
  --query "SELECT * FROM orders WHERE OrderDate >= '2024-01-01'" `
  --directory "C:\exports" `
  --fileoutput "orders.parquet" `
  --parquetcompression Snappy

Linux

./FastTransfer \
  --connectiontype mssql \
  --server "localhost" \
  --database "sales" \
  --trusted \
  --query "SELECT * FROM orders WHERE OrderDate >= '2024-01-01'" \
  --directory "/exports" \
  --fileoutput "orders.parquet" \
  --parquetcompression Snappy