-
Notifications
You must be signed in to change notification settings - Fork 5k
Implementing lightweight detection for new write operations #52977
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR implements lightweight detection for write operations to optimize the scaling logic performance. Instead of parsing all logs to extract blob paths, it only checks for the presence of write operations to trigger scaling events.
- Adds a new
HasBlobWritesAsync
method that efficiently detects write operations without full log parsing - Updates the scaling monitor to use the lightweight detection method
- Removes unused test field and improves logging messages
Reviewed Changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
File | Description |
---|---|
BlobLogListener.cs | Adds new HasBlobWritesAsync method for lightweight write operation detection |
BlobScalerMonitorProvider.cs | Updates scaling logic to use the new detection method and removes unused field |
BlobLogListenerTests.cs | Adds comprehensive test coverage for the new HasBlobWritesAsync method |
.../Microsoft.Azure.WebJobs.Extensions.Storage.Blobs/src/Listeners/BlobScalerMonitorProvider.cs
Outdated
Show resolved
Hide resolved
...age/Microsoft.Azure.WebJobs.Extensions.Storage.Blobs/tests/Listeners/BlobLogListenerTests.cs
Outdated
Show resolved
Hide resolved
...age/Microsoft.Azure.WebJobs.Extensions.Storage.Blobs/tests/Listeners/BlobLogListenerTests.cs
Show resolved
Hide resolved
sdk/storage/Microsoft.Azure.WebJobs.Extensions.Storage.Blobs/src/Listeners/BlobLogListener.cs
Outdated
Show resolved
Hide resolved
.../Microsoft.Azure.WebJobs.Extensions.Storage.Blobs/src/Listeners/BlobScalerMonitorProvider.cs
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good. This comment I had was just something you didn't know about beforehand, since this change is something that recently happened on the Azure.Storage.Blobs package side.
This PR introduces a streamlined approach to zero-to-one scaling logic within the Azure SDK. The current implementation reads and parses all classic monitoring logs in the $log container to extract modified blob paths. However, for scale-out from zero to one, this level of detail is unnecessary.
The updated logic simplifies the process by checking only for the presence of write operations within the last two hours—without downloading or parsing the full log content. This optimization reduces memory consumption and improves performance, especially in environments with high write volumes.