feat: upgrade AWS SDK from v1 to v2 for S3 client

[corylanou]

Summary

Migrates the S3 replica client from AWS SDK for Go v1 to v2, implementing modern Go patterns and best practices while maintaining backward compatibility.

Key Changes

Core Migration

• Update dependencies from aws-sdk-go v1 to aws-sdk-go-v2 v1.37.1
• Migrate client initialization to use config.LoadDefaultConfig()
• Update all S3 API calls to use v2 request/response types
• Implement proper error handling with smithy.APIError
• Add adaptive retry mode with 10 attempts for better resilience
• Update pagination to use built-in v2 paginators
• Remove unused errgroup dependency

Enhanced Features

• Bucket Validation: Add required bucket name validation in Init()
• Configurable Uploads: Add PartSize and Concurrency configuration for S3 multipart uploads
• Improved Error Handling: Add contextual error messages throughout the codebase
• Better Upload Validation: Check for ETag to confirm successful uploads
• 24-Hour Timeout: Set HTTP client timeout to 24 hours for long-running operations
• User-Agent Telemetry: Add "litestream" User-Agent header for better tracking
• Default Credential Chain: Full support for AWS IAM roles and credential providers

Code Quality Improvements (Latest)

• Simplified HTTP client creation to eliminate duplication
• Extracted configureEndpoint() helper method for DRY principle
• Consistently use DefaultRegion constant instead of hardcoded strings
• Added comprehensive test coverage for all new features

Testing Notes

Checksum Validation Behavior

During testing, we observed different checksum validation behaviors:

1. With moto mock server (CI):

   • Multipart uploads fail with checksum validation errors due to moto issue #8762
   • Moto doesn't add the -X suffix to composite checksums, causing validation to fail
   • Workaround: Test uses 4MB files to avoid multipart uploads when running against moto
   • Single-part uploads show: WARN Response has no supported checksum. Not validating response payload.

2. With real AWS S3 (manual integration tests):

   • Multipart uploads succeed but show: WARN Skipped validation of multipart checksum.
   • This warning appears when the SDK detects a composite checksum (format: <checksum>-<parts>)
   • The SDK appears to skip validation on download for multipart objects by default

Understanding AWS SDK v2 Checksum Behavior

Default Behavior (since v1.73.0):

• The SDK automatically calculates CRC32 checksums for all uploads
• For multipart uploads, checksums are calculated for each part
• The final checksum is a "checksum of checksums" rather than of the full object

Multipart Checksum Format:

• Composite checksums have format: <checksum-of-checksums>-<number-of-parts> (e.g., "DUoRhQ==-3")
• These differ from single-part checksums which represent the entire object

Validation on Download:

• The SDK warning suggests it skips validation for composite checksums on GetObject
• This may be expected behavior since composite checksums cannot be validated the same way as full-object checksums
• Data integrity is still ensured through:
  • Per-part checksums during upload
  • S3's internal validation during CompleteMultipartUpload
  • Our tests verify integrity with byte-level comparison

Note: There's an open issue (#3007) about the SDK always adding integrity checks to multipart uploads, which can cause compatibility issues with some S3-compatible services.

Benefits

• Better performance with improved connection pooling
• Modern context-based cancellation patterns
• Stronger type safety with dedicated types package
• Active maintenance and support from AWS
• Improved retry logic with adaptive mode
• More robust error handling and validation
• 24-hour timeout for long-running operations
• Better telemetry with User-Agent header
• Simplified credential management with default chain support

Testing

• All unit tests pass ✅
• Added comprehensive tests for isNotExists function ✅
• Added tests for bucket validation ✅
• Added tests for uploader configuration (adaptive for moto vs real S3) ✅
• Added tests for endpoint configuration helper ✅
• Added tests for HTTP client configuration ✅
• Added tests for credential configuration ✅
• Added tests for DefaultRegion constant usage ✅
• All linters pass (go vet, goimports, staticcheck) ✅
• Maintains compatibility with S3-compatible services (MinIO, DigitalOcean Spaces, Backblaze B2, etc.) ✅

Compatibility

• Fully backward compatible with existing configurations
• Supports all existing S3-compatible storage providers
• No breaking changes to the public API
• Enhanced support for AWS IAM roles (EC2, ECS, EKS)

Related Issues

• Fixes Upgrade AWS S3 SDK from v1 to v2 #674
• Created Update S3 multipart upload test once moto issue #8762 is resolved #700 to track updating tests once moto issue is resolved

Documentation Updates Needed

See comment below for comprehensive list of documentation updates needed after merge.

[github-actions]

✅ Manual integration tests have been run by @corylanou

View test results

[github-actions]

✅ Manual integration tests have been run by @corylanou

View test results

[corylanou]

Additional Improvements Added

Based on insights from PR #577 (Azure SDK upgrade), I've added several enhancements to further improve the AWS SDK v2 implementation:

✅ Improvements Added in Latest Commit:

1. User-Agent/Telemetry Support: Added custom User-Agent header "litestream" to all S3 requests using middleware, similar to Azure's ApplicationID approach for better telemetry tracking.

2. 24-Hour Timeout Configuration: Set HTTP client timeout to 24 hours for all operations, matching Azure's approach for long-running operations. This applies to both custom (with SkipVerify) and default HTTP clients.

3. Enhanced Retry Configuration: Increased retry attempts from 3 to 10 with adaptive retry mode for better resilience in production environments.

4. Documented Default Credential Chain: Added comprehensive comments explaining the AWS default credential chain that's automatically used when no explicit credentials are provided:

   • Environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
   • Shared credentials file (~/.aws/credentials)
   • EC2 Instance Profile credentials
   • ECS Task Role credentials
   • Web Identity Token credentials (for EKS with IRSA)

These changes bring the AWS SDK v2 implementation to parity with the Azure SDK's robustness and documentation standards.

All tests continue to pass, and all linters (go vet, goimports, staticcheck) run successfully.

[corylanou]

Latest Updates

Code Quality Improvements ✅

• Simplified HTTP client creation to eliminate duplication
• Extracted configureEndpoint() helper method for DRY principle
• Consistently use DefaultRegion constant instead of hardcoded strings

Test Coverage Added ✅

• Added comprehensive tests for all new features:
  • Endpoint configuration helper
  • HTTP client timeout (24 hours)
  • Credential configuration (static vs default chain)
  • Default region constant usage
• All tests passing with 100% success rate

---

Documentation Updates Needed After Merge 📚

The following documentation should be updated to reflect the AWS SDK v2 migration and new features:

1. Configuration Examples

Update S3 configuration examples to highlight:

• Default credential chain support (no credentials needed for EC2/ECS/EKS)
• New upload configuration options (part-size, concurrency)

2. AWS Authentication Methods

Document the supported authentication methods in order of precedence:

1. Explicit credentials in config (access-key-id, secret-access-key)
2. Environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
3. Shared credentials file (~/.aws/credentials)
4. EC2 Instance Profile (for EC2 instances)
5. ECS Task Role (for ECS/Fargate)
6. Web Identity Token (for EKS with IRSA)

3. Performance Tuning

Document the new upload configuration options:

replicas:
  - type: s3
    bucket: my-bucket
    part-size: 10485760  # 10MB parts for multipart uploads (default: 5MB)
    concurrency: 10      # Number of concurrent upload threads (default: 5)

4. Timeout and Retry Behavior

Document the improved resilience:

• 24-hour timeout for long-running operations
• Adaptive retry mode with up to 10 attempts
• Automatic handling of transient failures

5. Migration Notes

For users upgrading from previous versions:

• No breaking changes to configuration
• Improved performance with connection pooling
• Better error messages with context
• Automatic checksum validation (with noted limitations for multipart)

6. S3-Compatible Services

Confirm compatibility with:

• AWS S3 (all regions)
• MinIO
• DigitalOcean Spaces
• Backblaze B2
• Scaleway Object Storage
• Filebase
• Any S3-compatible API

7. Troubleshooting Section

Add notes about:

• Checksum validation warnings (expected for multipart uploads)
• How to enable debug logging if needed
• Common credential chain issues and solutions

Example Configuration Updates

Minimal config (using IAM role on EC2/ECS):

dbs:
  - path: /var/lib/myapp/db.sqlite
    replicas:
      - type: s3
        bucket: my-backup-bucket
        path: myapp
        region: us-west-2
        # No credentials needed - uses IAM role

Performance-optimized config:

dbs:
  - path: /var/lib/myapp/db.sqlite
    replicas:
      - type: s3
        bucket: my-backup-bucket
        path: myapp
        region: us-west-2
        part-size: 20971520  # 20MB for faster uploads on good connections
        concurrency: 20      # More parallel uploads

MinIO/S3-compatible config:

dbs:
  - path: /var/lib/myapp/db.sqlite
    replicas:
      - type: s3
        endpoint: http://minio.local:9000
        bucket: my-bucket
        path: myapp
        force-path-style: true
        access-key-id: minioadmin
        secret-access-key: minioadmin

These documentation updates will help users take full advantage of the AWS SDK v2 improvements and understand the new configuration options available.

[github-actions]

✅ Manual integration tests have been run by @corylanou

View test results