Added MultiDbClient support with OSS Cluster API #3734
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…v-active-active-pipeline
…ement, fixed data structure
…v-active-active-pipeline
… into vv-pubsub-support
… into vv-pubsub-support
|
It seems that the pubsub changes for the standalone are part of this change together with the newly added sharded_pubsub changes and are causing conflicts. Other than that, the changes are looking good. |
| indirect=True | ||
| ) | ||
| @pytest.mark.timeout(50) | ||
| def test_pubsub_failover_to_another_db(self, r_multi_db, fault_injector_client): |
There was a problem hiding this comment.
Do you plan on doing multi? Idk if it's needed for the AA, but it might be helpful to discover issues? Maybe some blocking command. (I've not the same AA context as you, so feel free to ignore this comment.)
There was a problem hiding this comment.
I have tests for transactions
vladvildanov
added a commit
that referenced
this pull request
Oct 7, 2025
* MultiDbClient implementation (#3696) * Added Database, Healthcheck, CircuitBreaker, FailureDetector * Added DatabaseSelector, exceptions, refactored existing entities * Added MultiDbConfig * Added DatabaseConfig * Added DatabaseConfig test coverage * Renamed DatabaseSelector into FailoverStrategy * Added CommandExecutor * Updated healthcheck to close circuit on success * Added thread-safeness * Added missing thread-safeness * Added missing thread-safenes for dispatcher * Refactored client to keep databases in WeightedList * Added database CRUD operations * Added on-fly configuration * Added background health checks * Added background healthcheck + half-open event * Refactored background scheduling * Refactored healthchecks * Removed code repetitions, fixed weight assignment, added loops enhancement, fixed data structure * Refactored configuration * Refactored failure detector * Refactored retry logic * Added scenario tests * Added pybreaker optional dependency * Added pybreaker to dev dependencies * Rename tests directory * Remove redundant checks * Handle retries if default is not set * Removed all Sentinel related * Added support for Pipeline and transactions (#3707) * Added Database, Healthcheck, CircuitBreaker, FailureDetector * Added DatabaseSelector, exceptions, refactored existing entities * Added MultiDbConfig * Added DatabaseConfig * Added DatabaseConfig test coverage * Renamed DatabaseSelector into FailoverStrategy * Added CommandExecutor * Updated healthcheck to close circuit on success * Added thread-safeness * Added missing thread-safeness * Added missing thread-safenes for dispatcher * Refactored client to keep databases in WeightedList * Added database CRUD operations * Added on-fly configuration * Added background health checks * Added background healthcheck + half-open event * Refactored background scheduling * Added support for Active-Active pipeline * Refactored healthchecks * Added Pipeline testing * Added support for transactions * Removed code repetitions, fixed weight assignment, added loops enhancement, fixed data structure * Added missing doc blocks * Refactored configuration * Refactored failure detector * Refactored retry logic * Added scenario tests * Added pybreaker optional dependency * Added pybreaker to dev dependencies * Rename tests directory * Added scenario tests for Pipeline and Transaction * Added handling of ConnectionRefusedError, added timeouts so cluster could recover * Increased timeouts * Refactored integration tests * Fixed property name * Removed sentinels * Removed unused method * Added support for Pub/Sub mode in MultiDbClient (#3722) * Added Database, Healthcheck, CircuitBreaker, FailureDetector * Added DatabaseSelector, exceptions, refactored existing entities * Added MultiDbConfig * Added DatabaseConfig * Added DatabaseConfig test coverage * Renamed DatabaseSelector into FailoverStrategy * Added CommandExecutor * Updated healthcheck to close circuit on success * Added thread-safeness * Added missing thread-safeness * Added missing thread-safenes for dispatcher * Refactored client to keep databases in WeightedList * Added database CRUD operations * Added on-fly configuration * Added background health checks * Added background healthcheck + half-open event * Refactored background scheduling * Added support for Active-Active pipeline * Refactored healthchecks * Added Pipeline testing * Added support for transactions * Removed code repetitions, fixed weight assignment, added loops enhancement, fixed data structure * Added missing doc blocks * Added support for Pub/Sub in MultiDBClient * Refactored configuration * Refactored failure detector * Refactored retry logic * Added scenario tests * Added pybreaker optional dependency * Added pybreaker to dev dependencies * Rename tests directory * Added scenario tests for Pipeline and Transaction * Added handling of ConnectionRefusedError, added timeouts so cluster could recover * Increased timeouts * Refactored integration tests * Added scenario tests for Pub/Sub * Updated healthcheck retry * Increased timeout to avoid unprepared state before tests * Added backoff retry and changed timeouts * Added retry for healthchecks to avoid fluctuations * Changed retry configuration for healthchecks * Fixed property name * Added check for thread results * Refactored docblocks (#3744) * Refactored healthcheck and failure detector to extend default one (#3747) * Added MultiDbClient support with OSS Cluster API (#3734) * Added Database, Healthcheck, CircuitBreaker, FailureDetector * Added DatabaseSelector, exceptions, refactored existing entities * Added MultiDbConfig * Added DatabaseConfig * Added DatabaseConfig test coverage * Renamed DatabaseSelector into FailoverStrategy * Added CommandExecutor * Updated healthcheck to close circuit on success * Added thread-safeness * Added missing thread-safeness * Added missing thread-safenes for dispatcher * Refactored client to keep databases in WeightedList * Added database CRUD operations * Added on-fly configuration * Added background health checks * Added background healthcheck + half-open event * Refactored background scheduling * Added support for Active-Active pipeline * Refactored healthchecks * Added Pipeline testing * Added support for transactions * Removed code repetitions, fixed weight assignment, added loops enhancement, fixed data structure * Added missing doc blocks * Added support for Pub/Sub in MultiDBClient * Refactored configuration * Refactored failure detector * Refactored retry logic * Added scenario tests * Added pybreaker optional dependency * Added pybreaker to dev dependencies * Rename tests directory * Added scenario tests for Pipeline and Transaction * Added handling of ConnectionRefusedError, added timeouts so cluster could recover * Increased timeouts * Refactored integration tests * Added scenario tests for Pub/Sub * Updated healthcheck retry * Increased timeout to avoid unprepared state before tests * Added backoff retry and changed timeouts * Added retry for healthchecks to avoid fluctuations * Changed retry configuration for healthchecks * Fixed property name * Added check for thread results * Added MultiDbClient support with OSS Cluster API * Removed database statuses * Increased test timeouts * Increased retry timeout * Increased timeout retries * Updated base threshold for retries * Fixed flacky tests * Added missing docblocks * Added LagAwareHealthCheck for MultiDBClient (#3737) * Added LagAwareHealthcheck * Added testing for LagAwareHealthCheck * Fixed timeouts * Added lag tollerance parameter * Decreased messages_count due to increased timeouts * Added docblocks * Added missing type hints * Fixed url * Refactored tests, URL and cluster support * Use primary node to send an API request * Added comment about RE bug * Moved None type to the beginning * Added health_check_url property to Database class * Added lag_aware_tolerance parameter to LagAwareHealthcheck (#3752) * Extract additional interfaces and abstract classes (#3754) * Added async implementation of MultiDBClient (#3762) * Extract additional interfaces and abstract classes * Added base async components * Added command executor * Added recurring background tasks with event loop only * Added MultiDBClient * Added scenario and config tests * Update redis/asyncio/multidb/healthcheck.py Co-authored-by: Copilot <[email protected]> * Update tests/test_asyncio/test_scenario/test_active_active.py Co-authored-by: Copilot <[email protected]> --------- Co-authored-by: Copilot <[email protected]> * Added pipeline and transaction support for MultiDBClient (#3763) * Extract additional interfaces and abstract classes * Added base async components * Added command executor * Added recurring background tasks with event loop only * Added MultiDBClient * Added scenario and config tests * Added pipeline and transaction support for MultiDBClient * Updated scenario tests to check failover * Added pub/sub support for MultiDBClient (#3764) * Extract additional interfaces and abstract classes * Added base async components * Added command executor * Added recurring background tasks with event loop only * Added MultiDBClient * Added scenario and config tests * Added pipeline and transaction support for MultiDBClient * Added pub/sub support for MultiDBClient * Added check for couroutines methods for pub/sub * Added support for Lag-Aware Healthcheck and OSS Cluster API (#3768) * Extract additional interfaces and abstract classes * Added base async components * Added command executor * Added recurring background tasks with event loop only * Added MultiDBClient * Added scenario and config tests * Added pipeline and transaction support for MultiDBClient * Added pub/sub support for MultiDBClient * Added check for couroutines methods for pub/sub * Added OSS Cluster API support for MultiDBCLient * Added support for Lag-Aware Healthcheck and OSS Cluster API * Increased timeouts between tests * Fixed space * Refactored Healthcheck and Failover strategy logic (#3771) * Extract additional interfaces and abstract classes * Added base async components * Added command executor * Added recurring background tasks with event loop only * Added MultiDBClient * Added scenario and config tests * Added pipeline and transaction support for MultiDBClient * Added pub/sub support for MultiDBClient * Added check for couroutines methods for pub/sub * Added OSS Cluster API support for MultiDBCLient * Added support for Lag-Aware Healthcheck and OSS Cluster API * Increased timeouts between tests * [Sync] Refactored healthcheck * [Async] Refactored healthcheck * [Sync] Refactored Failover Strategy * [Async] Refactored Failover Strategy * Changed default values according to a design doc * [Async] Added Strategy Executor * [Sync] Added Strategy Executor * Apply comments * Removed redundant dependency * Fixed async tests * Increased lag-aware tolerance * Fixed typing issue, increase health_check_interval, added timeout handling * Decreased retry cap, increased failure delay * Fixed async teardown * Fixed tests * Added graceful connection closing, added graceful hc tasks termination * Make sure active connection will be disconnected on failover * Close cluster connection on failover * Refactored Failure Detector (#3775) * Decreased timeouts * Added missing fixture * Fixed None exception * Codestyle changes * Codestyle changes * Skip async scenario tests * Codestyle change * Fixed unused arguments * Refactored bg scheduler * Fixed tests * Fixed tests * Codestyle fixes * Reduce timeouts to avoid overlaping with healthcheck * Marked tests non-clsuter only * Update timeouts * Skip scenario tests * Updated timeouts * Increased timeout * Refactored tests * Codestyle changes * Added documentation for Active-Active (#3753) * Added Active-Active documentation page * Added documentation for Active-Active * Refactored docs * Refactored pipeline and transaction section * Updated docs * Extended list of words * Re-write documentation * Fixed spelling * Update docs/multi_database.rst Co-authored-by: Elena Kolevska <[email protected]> * Apply suggested comments * Fixed spelling * Update docs/multi_database.rst Co-authored-by: Elena Kolevska <[email protected]> * Update docs/multi_database.rst Co-authored-by: Elena Kolevska <[email protected]> * Update docs/multi_database.rst Co-authored-by: Elena Kolevska <[email protected]> --------- Co-authored-by: Elena Kolevska <[email protected]> * Refactor unstable tests * Marked tests as non-clustered * Codestyle changes * Skipped tests in validating workflow --------- Co-authored-by: Copilot <[email protected]> Co-authored-by: petyaslavova <[email protected]> Co-authored-by: Elena Kolevska <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pull Request check-list
Please make sure to review and check all of these items:
NOTE: these things are not required to open a PR and can be done
afterwards / while the PR is open.
Description of change
Please provide a description of the change here.