Cloud Connectivity Overview

FastTransfer provides native support for exporting data directly to major cloud storage platforms without requiring intermediate local storage.

Supported Platforms

Platform	Protocol	Authentication Methods
AWS S3	s3://	Credentials file, Environment variables, IAM roles
Azure Blob Storage	abs://	Azure CLI, Connection string, Access keys, Managed Identity
Azure Data Lake Gen2	abfss://	Azure CLI, Connection string, Access keys, Managed Identity
Google Cloud Storage	gs://	gcloud CLI, Service account, Default credentials
S3-Compatible Storage	s3://	Credentials file, Environment variables
OneLake	onelake://	Azure CLI, Connection string, Access keys, Managed Identity

Key Features

Direct Upload

Export data directly to cloud storage without local disk space requirements.

note

FastTransfer will use multi-part uploads for all cloud provider

Warning

Since FastTransfer can be cancelled, killed, or crash unexpectedly, any multipart uploads it initiates may remain incomplete. These unfinished multipart uploads continue to occupy storage and may incur additional costs. To prevent this, you should configure a lifecycle management rule on your bucket to automatically delete incomplete multipart uploads older than x days.

Multiple Authentication Methods

FastTransfer supports multiple authentication mechanisms, including CLI profiles, credentials files, environment variables, and cloud-native identity solutions. Authentication is resolved through a fallback chain, in the following order:

• The --cloudprofile option is used first, if provided.
• Standard authentication environment variables are used as the first fallback.
• Managed Identity is used as the second fallback.
• An IAM role/profile is used as the final fallback.

Profile Management

Use the --cloudprofile parameter can allow you to switch between different cloud accounts and environments (development, staging, production).

Automatic Retry

Built-in retry logic for network failures and transient cloud service errors.

Quick Start

1. Configure Cloud Authentication

Choose your cloud provider and follow the setup guide:

• AWS S3 Setup
• Azure Setup
• Google Cloud Storage Setup

2. Export to Cloud

Use the appropriate URI scheme in the --directory parameter:

# AWS S3 & S3-Compatible
./FastTransfer /
...
 --directory "s3://my-bucket/exports" /
 --fileoutput "data.parquet" /
...

# Azure Blob
./FastTransfer \
...
 --directory "abs://account.blob.core.windows.net/container/exports/" \
 --fileoutput "data.parquet" \
...

# Azure Datalake Gen 2
./FastTransfer \
...
 --directory "abs://account.dfs.core.windows.net/container/exports/" \
 --fileoutput "data.parquet" \
...

# Google Cloud Storage
./FastTransfer \
...
 --directory "gs://my-bucket/exports" \
 --fileoutput "data.parquet" \
...

# OneLake
./FastTransfer \
...
 --directory "onelake://MyWorSpace/MyLakeHouse/MyDir/" \
 --fileoutput "data.parquet" \
...

3. Use Profiles

Switch between different cloud accounts:

./FastTransfer \
...
 --directory "s3://prod-bucket/exports" \
 --cloudprofile "production" \
...


./FastTransfer \
...
 --directory "s3://dev-bucket/exports" \
 --cloudprofile "development"
...