Skip to main content

AWS S3 Configuration

Complete guide for configuring AWS S3 connectivity with FastTransfer.

Authentication Methods​

FastTransfer supports multiple AWS authentication methods, listed in order of precedence:

  1. AWS Credentials File (recommended for local development)
  2. Environment Variables (recommended for CI/CD)
  3. IAM Roles (recommended for AWS infrastructure)

1. AWS Credentials File​

The most common method for local development and testing.

Location:

  • Windows: C:\Users\YourName\.aws\credentials
  • Linux: ~/.aws/credentials

Credentials File Format:

[default]
aws_access_key_id = AKIAIOSFODNN7EXAMPLE
aws_secret_access_key = wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

[production]
aws_access_key_id = AKIAI44QH8DHBEXAMPLE
aws_secret_access_key = je7MtGbClwBF/2Zp9Utk/h3yCo8nvbEXAMPLEKEY

[development]
aws_access_key_id = AKIAI44QH8DHBEXAMPLE
aws_secret_access_key = je7MtGbClwBF/2Zp9Utk/h3yCo8nvbEXAMPLEKEY

Configuration File (Optional):

Location: ~/.aws/config

[default]
region = us-east-1
output = json

[profile production]
region = eu-west-1
output = json

[profile development]
region = us-west-2
output = json

Using Profiles with FastTransfer:

# Use default profile
./FastTransfer \
...
 --directory "s3://my-bucket/exports"  \
 --fileoutput "data.parquet" \
...

# Use production profile
./FastTransfer \
...
 --directory "s3://prod-bucket/exports" \
 --cloudprofile "production" \
 --fileoutput "data.parquet" \
...

# Use development profile
./FastTransfer \
...
 --directory "s3://dev-bucket/exports" \
 --cloudprofile "development" \
 --fileoutput "data.parquet" \
...

2. Environment Variables​

Useful for CI/CD pipelines and containerized environments.

# Set AWS credentials
$env:AWS_ACCESS_KEY_ID="AKIAIOSFODNN7EXAMPLE"
$env:AWS_SECRET_ACCESS_KEY="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
$env:AWS_DEFAULT_REGION="us-east-1"

# Optional: Session token for temporary credentials
$env:AWS_SESSION_TOKEN="FwoGZXIvYXdzEBYaD..."

# Run FastTransfer
.\\FastTransfer.exe `
...
 --directory "s3://my-bucket/exports" `
 --fileoutput "data.parquet" `
...

3. IAM Roles (EC2, ECS, Lambda)​

When FastTransfer runs on AWS infrastructure, it can use automatically the instance's IAM role. No credentials configuration needed!

Supported AWS Services:

  • Amazon EC2 instances
  • Amazon ECS tasks
  • AWS Lambda functions
  • AWS Batch jobs
  • Amazon EKS pods (with IAM roles for service accounts)

Example IAM Policy for FastTransfer:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:PutObjectAcl",
        "s3:GetObject"
      ],
      "Resource": "arn:aws:s3:::my-bucket/exports/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket"
      ],
      "Resource": "arn:aws:s3:::my-bucket"
    }
  ]
}

S3 URI Format​

s3://bucket-name/path/to/directory/

URI Examples​

# Root of bucket
 --directory "s3://my-bucket"

# Single folder level
 --directory "s3://my-bucket/exports"

# Multiple folder levels
 --directory "s3://my-bucket/exports/sales/2024"

# Hive-style partitioning
 --directory "s3://my-bucket/data/year=2024/month=01/day=15"

Complete Examples​

Basic Export to S3​

.\\FastTransfer.exe `
  --connectiontype "mssql" `
  --server "localhost" `
  --database "SalesDB" `
  --trusted `
  --sourceschema "dbo" `
  --sourcetable "orders" `
  --directory "s3://my-bucket/exports/{sourcetable}" `
  --fileoutput "{sourcetable}.parquet"

Parallel Export with Profile​

.\\FastTransfer.exe `
  --connectiontype "mssql" `
  --server "prod-sql-01" `
  --database "SalesDB" `
  --trusted `
  --query "SELECT * FROM Sales.Orders WHERE OrderDate >= '2024-01-01'" `
  --directory "s3://prod-datalake/sales/orders" `
  --fileoutput "orders.parquet" `
  --parallelmethod "RangeId" `
  --distributekeycolumn "OrderID" `
  --paralleldegree 8 `
  --cloudprofile "production"

Export with Date Partitioning​

.\\FastTransfer.exe `
  --connectiontype "pgsql" `
  --server "localhost" `
  --port "5432" `
  --database "analytics" `
  --user "postgres" `
  --password "postgres" `
  --sourceschema "public" `
  --sourcetable "events" `
  --directory "s3://analytics-bucket/events/" `
  --fileoutput "events.parquet" `
  --parallelmethod "DataDriven" `
  --distributekeycolumn "eventmonth" `
  --paralleldegree 10

S3-Compatible Storage​

FastTransfer works with S3-compatible storage services like MinIO, DigitalOcean Spaces, Wasabi, and OVH Object Storage.

Configuration for S3-Compatible Services​

# Set endpoint URL for S3-compatible service
$env:AWS_ACCESS_KEY_ID="your-access-key"
$env:AWS_SECRET_ACCESS_KEY="your-secret-key"
$env:S3_ENDPOINT="https://s3.your-provider.com"

# For MinIO
$env:S3_ENDPOINT="http://localhost:9000"

Required Permissions​

Minimum Permissions​

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject"
      ],
      "Resource": "arn:aws:s3:::my-bucket/exports/*"
    }
  ]
}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:PutObjectAcl",
        "s3:GetObject",
        "s3:DeleteObject"
      ],
      "Resource": "arn:aws:s3:::my-bucket/exports/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket"
      ],
      "Resource": "arn:aws:s3:::my-bucket",
      "Condition": {
        "StringLike": {
          "s3:prefix": "exports/*"
        }
      }
    }
  ]
}
Copyright © 2026 Architecture & Performance. Built with Docusaurus.