Cloud Options

FastTransfer supports direct exports to cloud storage platforms: AWS S3, Azure Blob Storage, Azure Data Lake Gen2, and Google Cloud Storage. For detailed cloud authentication setup and configuration, see the Cloud Connectivity documentation.

note

FastTransfer will use multi-part uploads for all cloud provider

Warning

Since FastTransfer can be cancelled, killed, or crash unexpectedly, any multipart uploads it initiates may remain incomplete. These unfinished multipart uploads continue to occupy storage and may incur additional costs. To prevent this, you should configure a lifecycle management rule on your bucket to automatically delete incomplete multipart uploads older than x days.

Supported Cloud Platforms

To export data to a cloud storage service, use the URI scheme specific to your target destination with the --directory value

Platform	URI Scheme	Example
AWS S3	s3://	s3://my-bucket/data/
Azure Blob	abs://	abs://myaccount.blob.core.windows.net/mycontainer/mydir/
Azure Data Lake Gen2	abfss://	abfss://myaccount.dfs.core.windows.net/mycontainer/mydir/
Google Cloud Storage	gs://	gs://my-bucket/data/
S3-Compatible	s3://	s3://mybucket/export/ (MinIO, OVH, etc.)
OneLake	onelake://	onelake://MyWorSpace/MyLakeHouse/MyDir/

Cloud Profile

You can use the --cloudprofile parameter to specify which cloud credentials profile to use.

./FastTransfer \
...
 --directory "s3://my-bucket/exports" \
 --fileoutput "orders.parquet" \
 --cloudprofile "production" \
...

Syntax:

• Long form: --cloudprofile <profile_name>