Upgrade Azure Storage SDK, allow use of default credential #577
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@benbjohnson that is a good patch. Essentially, it allows Litestream to function with the workload identity, so operations won't need to rotate the key and could utilise the OIDC from Microsoft Entra in K8s or VM deployments. |
40 tasks
corylanou
added a commit
that referenced
this pull request
Aug 13, 2025
Inspired by PR #577 (Azure SDK upgrade), this commit adds several improvements to the AWS SDK v2 implementation: - Add User-Agent header 'litestream' for telemetry tracking - Set 24-hour timeout for long-running operations (matches Azure approach) - Increase retry attempts from 3 to 10 with adaptive retry mode - Document AWS default credential chain support - Ensure consistent HTTP client timeout configuration These changes improve resilience, observability, and documentation while maintaining compatibility with various AWS authentication methods. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
corylanou
added a commit
that referenced
this pull request
Aug 14, 2025
Migrates Azure Blob Storage client from deprecated track 1 SDK to modern track 2, implementing authentication improvements and patterns consistent with the AWS SDK v2 upgrade. ## Key Changes ### SDK Migration - Replace deprecated azure-storage-blob-go v0.15.0 with azure-sdk-for-go/sdk/storage/azblob v1.4.1 - Remove dependency on deprecated Azure pipeline packages - Add azidentity for default credential chain support ### Authentication Enhancements - Maintain existing shared key authentication (AccountName + AccountKey) - Add Azure default credential chain support (similar to AWS SDK): - Environment variables (AZURE_CLIENT_ID, AZURE_CLIENT_SECRET, AZURE_TENANT_ID) - Managed Identity (for Azure VMs, App Service, Functions, AKS) - Azure CLI credentials - Visual Studio Code credentials - Preserve LITESTREAM_AZURE_ACCOUNT_KEY environment variable support ### Implementation Updates - Convert from ContainerURL to modern Client pattern - Update all blob operations to use new SDK methods: - UploadStream for writing - DownloadStream for reading - DeleteBlob for deletion - NewListBlobsFlatPager for listing - Maintain 24-hour timeout for long-running operations - Add retry configuration with 10 attempts and adaptive backoff - Include "litestream" application ID for telemetry ### Iterator Consistency with S3 - Remove goroutine/channel-based iterator approach - Implement direct pagination in Next() method (matching S3 pattern) - Simplify state management with pager interface - Maintain metrics integration and error handling ### Error Handling - Update to use azcore.ResponseError for error checking - Use bloberror package constants for specific error types - Maintain backward compatibility for os.ErrNotExist ## Benefits - Modern SDK with active support and regular updates - Enhanced authentication options for better security - Improved performance and reliability - Better integration with Azure services - Consistent patterns across storage backends (S3 and Azure) - Future-proof for Azure authentication enhancements ## Testing - All Go compilation checks pass - All linters pass (go vet, staticcheck) - Code formatting verified - Maintains backward compatibility with existing configurations Related to #459, #577 Fixes compatibility issues with deprecated Azure SDK 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
6 tasks
|
Thank you for rebasing #459! We appreciate the effort to keep this upgrade moving forward. We've created a new implementation in #709 with:
Closing in favor of #709. Thanks for helping push this forward! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is mostly just a rebased version of #459
I also add support for the default Azure credential flow
Co-authored-by: ItalyPaleAle