AWS Lambda Docker Wrapper

The AWS Lambda Docker wrapper enables you to run FastTransfer as a serverless function in AWS Lambda, allowing you to execute database transfers on-demand or on a schedule without managing infrastructure.

Overview

This wrapper packages FastTransfer into a Docker container optimized for AWS Lambda, enabling:

Benefits:

• Serverless execution: No servers to manage
• Event-driven: Trigger transfers from S3 events, SNS, SQS, EventBridge, etc.
• Scheduled transfers: Use EventBridge rules for cron-like scheduling
• Auto-scaling: Lambda scales automatically with demand
• Cost-effective: Pay only for execution time
• VPC support: Connect to databases in private subnets

Use Cases:

• Scheduled ETL jobs
• Event-driven data synchronization
• On-demand data extracts triggered by API Gateway
• Cloud-to-cloud database migrations

Repository

The Lambda wrapper source code and deployment templates are available on GitHub:

🔗 FastTransfer Lambda Wrapper on GitHub

Architecture

┌─────────────────┐      ┌──────────────────┐      ┌─────────────────┐
│   Trigger       │      │   AWS Lambda     │      │  Target DB      │
│  (EventBridge,  │─────▶│  (FastTransfer   │─────▶│  (RDS, Aurora,  │
│   API Gateway,  │      │   Container)     │      │   External)     │
│   S3, etc.)     │      └──────────────────┘      └─────────────────┘
└─────────────────┘              │
                                 │
                                 ▼
                         ┌──────────────────┐
                         │   Source DB      │
                         │  (RDS, Aurora,   │
                         │   External)      │
                         └──────────────────┘

Requirements

AWS Account Setup

• AWS Account with appropriate permissions
• AWS CLI configured
• Docker installed locally
• ECR repository for Docker images

Lambda Configuration

• Memory: 1024 MB minimum, 3008 MB recommended for large transfers
• Timeout: 15 minutes maximum (Lambda limit)
• Ephemeral storage: 512 MB minimum, up to 10 GB for large datasets
• VPC: Configure if databases are in private subnets

Lambda Timeout Limit

AWS Lambda has a maximum execution time of 15 minutes. For transfers that take longer, consider using AWS Batch, ECS, or EC2 instead.

Installation

Step 1: Clone the Repository

git clone https://github.com/aetperf/FastTransfer-Lambda-Wrapper.git
cd FastTransfer-Lambda-Wrapper

Step 2: Build Docker Image

The repository includes a Dockerfile optimized for Lambda:

FROM public.ecr.aws/lambda/python:3.11

# Install FastTransfer
COPY FastTransfer ${LAMBDA_TASK_ROOT}/FastTransfer
RUN chmod +x ${LAMBDA_TASK_ROOT}/FastTransfer

# Copy wrapper code
COPY lambda_function.py ${LAMBDA_TASK_ROOT}
COPY requirements.txt ${LAMBDA_TASK_ROOT}

# Install Python dependencies
RUN pip install -r requirements.txt

# Set the CMD to your handler
CMD [ "lambda_function.handler" ]

Build the image:

docker build -t fasttransfer-lambda .

Step 3: Push to Amazon ECR

# Create ECR repository
aws ecr create-repository --repository-name fasttransfer-lambda --region us-east-1

# Login to ECR
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin <account-id>.dkr.ecr.us-east-1.amazonaws.com

# Tag image
docker tag fasttransfer-lambda:latest <account-id>.dkr.ecr.us-east-1.amazonaws.com/fasttransfer-lambda:latest

# Push image
docker push <account-id>.dkr.ecr.us-east-1.amazonaws.com/fasttransfer-lambda:latest

Step 4: Create Lambda Function

Using AWS Console:

1. Go to Lambda → Create Function
2. Choose "Container image"
3. Select your ECR image
4. Configure memory (1024 MB+) and timeout (5-15 minutes)
5. Attach IAM role with necessary permissions

Using AWS CLI:

aws lambda create-function \
  --function-name fasttransfer-function \
  --package-type Image \
  --code ImageUri=<account-id>.dkr.ecr.us-east-1.amazonaws.com/fasttransfer-lambda:latest \
  --role arn:aws:iam::<account-id>:role/lambda-execution-role \
  --timeout 900 \
  --memory-size 2048 \
  --ephemeral-storage Size=1024

Step 5: Configure VPC (if needed)

If your databases are in private subnets:

aws lambda update-function-configuration \
  --function-name fasttransfer-function \
  --vpc-config SubnetIds=subnet-xxxxx,subnet-yyyyy,SecurityGroupIds=sg-zzzzz

Usage

Event Structure

The Lambda function expects JSON event with transfer parameters:

{
  "source": {
    "type": "pgsql",
    "server": "source-db.cluster-xxxxx.us-east-1.rds.amazonaws.com",
    "user": "pguser",
    "password": "pgpass",
    "database": "sales",
    "table": "orders"
  },
  "target": {
    "type": "mssql",
    "server": "target-db.xxxxx.us-east-1.rds.amazonaws.com",
    "user": "sqluser",
    "password": "sqlpass",
    "database": "warehouse",
    "table": "orders"
  },
  "options": {
    "loadMode": "Append",
    "degree": 4,
    "batchSize": 10000
  }
}

Invoke from AWS CLI

aws lambda invoke \
  --function-name fasttransfer-function \
  --payload file://transfer-config.json \
  response.json

cat response.json

Invoke from Python (boto3)

import boto3
import json

lambda_client = boto3.client('lambda', region_name='us-east-1')

payload = {
    "source": {
        "type": "pgsql",
        "server": "source-db.cluster-xxxxx.us-east-1.rds.amazonaws.com",
        "user": "pguser",
        "password": "pgpass",
        "database": "sales",
        "table": "orders"
    },
    "target": {
        "type": "mssql",
        "server": "target-db.xxxxx.us-east-1.rds.amazonaws.com",
        "user": "sqluser",
        "password": "sqlpass",
        "database": "warehouse",
        "table": "orders"
    }
}

response = lambda_client.invoke(
    FunctionName='fasttransfer-function',
    InvocationType='RequestResponse',
    Payload=json.dumps(payload)
)

result = json.loads(response['Payload'].read())
print(f"Status: {result['statusCode']}")
print(f"Result: {result['body']}")

Scheduled Transfer with EventBridge

Create a rule that triggers the Lambda function daily:

# Create EventBridge rule
aws events put-rule \
  --name daily-transfer-2am \
  --schedule-expression "cron(0 2 * * ? *)" \
  --description "Daily database transfer at 2 AM UTC"

# Add Lambda as target
aws events put-targets \
  --rule daily-transfer-2am \
  --targets "Id"="1","Arn"="arn:aws:lambda:us-east-1:<account-id>:function:fasttransfer-function","Input"='{
    "source": {"type": "pgsql", "server": "source-db.us-east-1.rds.amazonaws.com", "database": "sales", "table": "orders"},
    "target": {"type": "mssql", "server": "target-db.us-east-1.rds.amazonaws.com", "database": "warehouse", "table": "orders"}
  }'

# Grant EventBridge permission to invoke Lambda
aws lambda add-permission \
  --function-name fasttransfer-function \
  --statement-id EventBridgeInvoke \
  --action lambda:InvokeFunction \
  --principal events.amazonaws.com \
  --source-arn arn:aws:events:us-east-1:<account-id>:rule/daily-transfer-2am

API Gateway Integration

Expose FastTransfer as a REST API:

# Create REST API
aws apigateway create-rest-api \
  --name fasttransfer-api \
  --description "FastTransfer API"

# Configure Lambda integration
# (See AWS documentation for complete API Gateway setup)

Then invoke via HTTP:

curl -X POST https://xxxxxx.execute-api.us-east-1.amazonaws.com/prod/transfer \
  -H "Content-Type: application/json" \
  -d '{
    "source": {"type": "pgsql", "server": "source.rds.amazonaws.com", "database": "db", "table": "table"},
    "target": {"type": "mssql", "server": "target.rds.amazonaws.com", "database": "db", "table": "table"}
  }'

Credential Management

AWS Secrets Manager

Store database credentials securely:

import boto3
import json

def get_credentials(secret_name):
    client = boto3.client('secretsmanager', region_name='us-east-1')
    response = client.get_secret_value(SecretId=secret_name)
    return json.loads(response['SecretString'])

# In lambda_function.py
source_creds = get_credentials('fasttransfer/source-db')
target_creds = get_credentials('fasttransfer/target-db')

payload = {
    "source": {
        "type": "pgsql",
        "server": source_creds['host'],
        "user": source_creds['username'],
        "password": source_creds['password'],
        "database": source_creds['database'],
        "table": "orders"
    },
    "target": {
        "type": "mssql",
        "server": target_creds['host'],
        "user": target_creds['username'],
        "password": target_creds['password'],
        "database": target_creds['database'],
        "table": "orders"
    }
}

Create secrets:

aws secretsmanager create-secret \
  --name fasttransfer/source-db \
  --secret-string '{"host":"source.rds.amazonaws.com","username":"pguser","password":"pgpass","database":"sales"}'

aws secretsmanager create-secret \
  --name fasttransfer/target-db \
  --secret-string '{"host":"target.rds.amazonaws.com","username":"sqluser","password":"sqlpass","database":"warehouse"}'

IAM Roles and Permissions

Lambda execution role needs:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:*:*:*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "ec2:CreateNetworkInterface",
        "ec2:DescribeNetworkInterfaces",
        "ec2:DeleteNetworkInterface"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue"
      ],
      "Resource": "arn:aws:secretsmanager:*:*:secret:fasttransfer/*"
    }
  ]
}

Monitoring and Logging

CloudWatch Logs

Lambda automatically logs to CloudWatch:

import logging

logger = logging.getLogger()
logger.setLevel(logging.INFO)

def handler(event, context):
    logger.info(f"Starting transfer: {event}")
    
    # Execute FastTransfer
    result = execute_transfer(event)
    
    logger.info(f"Transfer completed: {result}")
    return result

CloudWatch Metrics

Create custom metrics:

import boto3

cloudwatch = boto3.client('cloudwatch')

def publish_metrics(row_count, duration):
    cloudwatch.put_metric_data(
        Namespace='FastTransfer',
        MetricData=[
            {
                'MetricName': 'RowsTransferred',
                'Value': row_count,
                'Unit': 'Count'
            },
            {
                'MetricName': 'TransferDuration',
                'Value': duration,
                'Unit': 'Seconds'
            }
        ]
    )

X-Ray Tracing

Enable AWS X-Ray for detailed tracing:

aws lambda update-function-configuration \
  --function-name fasttransfer-function \
  --tracing-config Mode=Active

Performance Optimization

Memory Configuration

More memory = more CPU power:

• 1024 MB: Small transfers (under 100K rows)
• 2048 MB: Medium transfers (under 1M rows)
• 3008 MB: Large transfers (1M+ rows)

Ephemeral Storage

Increase if working with large datasets:

aws lambda update-function-configuration \
  --function-name fasttransfer-function \
  --ephemeral-storage Size=2048  # Up to 10240 MB

Provisioned Concurrency

For consistent performance:

aws lambda put-provisioned-concurrency-config \
  --function-name fasttransfer-function \
  --provisioned-concurrent-executions 5

Limitations

Lambda Constraints

• Maximum execution time: 15 minutes
• Maximum memory: 10 GB
• Maximum ephemeral storage: 10 GB
• Maximum payload: 6 MB (synchronous), 256 KB (asynchronous)

Workarounds for Large Transfers

For transfers exceeding Lambda limits:

1. Split into batches: Transfer in chunks using queries with LIMIT/OFFSET
2. Use Step Functions: Orchestrate multiple Lambda invocations
3. Use AWS Batch: For longer-running transfers
4. Use ECS Fargate: For very large transfers

Cost Considerations

Lambda pricing based on:

• Invocations: $0.20 per 1M requests
• Duration: $0.0000166667 per GB-second
• Data transfer: Standard AWS data transfer rates

Example calculation:

• 100 daily transfers
• 2 GB memory, 5 minutes each
• Cost: ~$15/month for compute + data transfer

Compare with:

• EC2 t3.medium 24/7: ~$30/month
• RDS proxy: Additional costs

Further Information

For complete source code, deployment templates, and examples:

🔗 Visit the GitHub Repository

For AWS documentation:

• AWS Lambda with Container Images
• Lambda VPC Configuration
• EventBridge Scheduling

For FastTransfer documentation:

• CLI Parameters
• Getting Started
• Examples