Skip to main content

Google Cloud Storage Configuration

Complete guide for configuring Google Cloud Storage connectivity with FastTransfer.

Authentication Methods​

FastTransfer supports multiple Google Cloud authentication methods:

  1. gcloud CLI (recommended for local development)
  2. Service Account Key File (recommended for production)
  3. Default Application Credentials (for GCP infrastructure)

The simplest method for local development and testing.

Setup:

# Install gcloud CLI
# Windows: Download from https://cloud.google.com/sdk/docs/install
# Linux: curl https://sdk.cloud.google.com | bash

# Initialize gcloud
gcloud init

# Login to Google Cloud
gcloud auth login

# List available projects
gcloud projects list

# Set active project
gcloud config set project my-project-id

# Create application default credentials
gcloud auth application-default login

FastTransfer will automatically use the gcloud CLI's authentication credentials.

Using with FastTransfer:

# No additional configuration needed - just run FastTransfer
./FastTransfer \
...
 --directory "gs://my-bucket/exports" \
 --fileoutput "data.parquet" \
...

2. Service Account Key File​

Recommended for production environments, CI/CD pipelines, and automated workflows.

Create Service Account:

  1. Go to IAM & Admin → Service Accounts
  2. Click Create Service Account
  3. Enter name (e.g., "FastTransfer-export")
  4. Click Create and Continue
  5. Grant Storage Object Admin role
  6. Click Continue → Done
  7. Click on the service account
  8. Go to Keys tab
  9. Click Add Key → Create new key
  10. Select JSON format
  11. Click Create (JSON file will download)

Use Service Account Key:

# Set path to service account key file
$env:GOOGLE_APPLICATION_CREDENTIALS="C:\keys\FastTransfer-key.json"

# Run FastTransfer
.\\FastTransfer.exe `
...
 --directory "gs://my-bucket/exports" `
 --fileoutput "data.parquet" `
...

3. Default Application Credentials​

When FastTransfer runs on Google Cloud infrastructure, it automatically uses the default service account. No credentials configuration needed!

Supported GCP Services:

  • Compute Engine instances
  • Google Kubernetes Engine (GKE) pods
  • Cloud Run services
  • Cloud Functions
  • App Engine

Verify Default Credentials:

gcloud auth application-default print-access-token

GCS URI Format​

gs://bucket-name/path/to/directory/

URI Examples​

# Root of bucket
 --directory "gs://my-bucket"

# Single folder level
 --directory "gs://my-bucket/exports"

# Multiple folder levels
 --directory "gs://my-bucket/exports/sales/2024"

# Hive-style partitioning
 --directory "gs://my-bucket/data/year=2024/month=01/day=15"

Complete Examples​

Basic Export to GCS​

.\\FastTransfer.exe `
  --connectiontype "mssql" `
  --server "localhost" `
  --database "SalesDB" `
  --trusted `
  --sourceschema "dbo" `
  --sourcetable "Orders" `
  --directory "gs://my-bucket/exports/orders" `
  --fileoutput "orders.parquet"

Parallel Export with Query​

.\\FastTransfer.exe `
  --connectiontype "pgsql" `
  --server "localhost" `
  --port "5432" `
  --database "analytics" `
  --user "postgres" `
  --password "postgres" `
  --query "SELECT * FROM events WHERE event_date >= '2024-01-01'" `
  --directory "gs://analytics-bucket/events/2024" `
  --fileoutput "events.parquet" `
  --parallelmethod "Ctid" `
  --paralleldegree 10

Export with Date Partitioning​

.\\FastTransfer.exe `
  --connectiontype "mysql" `
  --server "localhost" `
  --port "3306" `
  --database "ecommerce" `
  --user "root" `
  --password "password" `
  --sourceschema "ecommerce" `
  --sourcetable "orders" `
  --directory "gs://datalake-bucket/orders/" `
  --fileoutput "orders.parquet" `
  --parallelmethod "DataDriven" `
  --distributekeycolumn "o_orderdate" `
  --paralleldegree 8

Export from Oracle to GCS using Rowid parallelmethod​

.\\FastTransfer.exe `
  --connectiontype "oraodp" `
  --server "localhost:1521/ORCLPDB1" `
  --user "FastUser" `
  --password "FastPassword" `
  --sourceschema "SALES" `
  --sourcetable "ORDERS" `
  --directory "gs://warehouse-bucket/oracle-export/orders" `
  --fileoutput "orders.parquet" `
  --parallelmethod "Rowid" `
  --paralleldegree 8

Bucket Configuration​

Create Bucket​

  1. Go to Cloud Storage → Buckets
  2. Click Create Bucket
  3. Enter bucket name (globally unique)
  4. Choose location type and region
  5. Choose storage class (Standard, Nearline, Coldline, Archive)
  6. Set access control (Uniform or Fine-grained)
  7. Click Create

Required Permissions​

IAM Roles​

Recommended Roles:

  • Storage Object Creator (roles/storage.objectCreator) - Create objects only
  • Storage Object Viewer (roles/storage.objectViewer) - Read objects only

Grant Access via Console​

  1. Go to Cloud Storage → Buckets
  2. Click bucket name
  3. Go to Permissions tab
  4. Click Grant Access
  5. Enter principal (user, service account, or group)
  6. Select role (e.g., Storage Object Admin)
  7. Click Save

Grant Access via gcloud CLI​


# Grant to service account
gsutil iam ch serviceAccount:FastTransfer-export@my-project-id.iam.gserviceaccount.com:roles/storage.objectAdmin gs://my-bucket

Minimal Permissions​

Minimum permissions for FastTransfer instead of bucket storage admin:

  • storage.objects.create - Upload objects
  • storage.objects.delete - Delete objects (for overwrite)
  • storage.objects.get - Read objects (for verification)
  • storage.objects.list - List objects in bucket
Copyright © 2026 Architecture & Performance. Built with Docusaurus.