optimize: use elasticsearch msearch API for batch queries #604
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit refactors the Elasticsearch
batch_query
to utilize the msearch (multi-search) API instead of
sending individual queries in a loop.
The msearch API allows sending multiple search requests in a single
HTTP call, which significantly reduces network overhead and improves
query throughput, especially when performing batch operations.
Changes include:
Dockerfile
batch_query
calls with msearch API inmodule.py
Attention:
thread_pool.search.queue_size=1000
(with thread_pool.search.size=1),allowing more requests to queue but still executing them in a single thread