Multi-database client implementation #3784

vladvildanov · 2025-10-02T07:50:47Z

Pull Request check-list

Please make sure to review and check all of these items:

Do tests and lints pass with this change?
Do the CI tests pass with this change (enable it first in your forked repo and wait for the github action build to finish)?
Is the new or changed code fully tested?
Is a documentation update included (if this change modifies existing APIs, or introduces new ones)?
Is there an example added to the examples folder (if applicable)?

NOTE: these things are not required to open a PR and can be done
afterwards / while the PR is open.

Description of change

Following feature provides a client to communicate with multiple eventually consistent database, whereas one active and N passive databases. Client is able to failover and fallback between databases in seamless way and provides a variety of configuration options depends on client application preferences

* Added Database, Healthcheck, CircuitBreaker, FailureDetector * Added DatabaseSelector, exceptions, refactored existing entities * Added MultiDbConfig * Added DatabaseConfig * Added DatabaseConfig test coverage * Renamed DatabaseSelector into FailoverStrategy * Added CommandExecutor * Updated healthcheck to close circuit on success * Added thread-safeness * Added missing thread-safeness * Added missing thread-safenes for dispatcher * Refactored client to keep databases in WeightedList * Added database CRUD operations * Added on-fly configuration * Added background health checks * Added background healthcheck + half-open event * Refactored background scheduling * Refactored healthchecks * Removed code repetitions, fixed weight assignment, added loops enhancement, fixed data structure * Refactored configuration * Refactored failure detector * Refactored retry logic * Added scenario tests * Added pybreaker optional dependency * Added pybreaker to dev dependencies * Rename tests directory * Remove redundant checks * Handle retries if default is not set * Removed all Sentinel related

* Added Database, Healthcheck, CircuitBreaker, FailureDetector * Added DatabaseSelector, exceptions, refactored existing entities * Added MultiDbConfig * Added DatabaseConfig * Added DatabaseConfig test coverage * Renamed DatabaseSelector into FailoverStrategy * Added CommandExecutor * Updated healthcheck to close circuit on success * Added thread-safeness * Added missing thread-safeness * Added missing thread-safenes for dispatcher * Refactored client to keep databases in WeightedList * Added database CRUD operations * Added on-fly configuration * Added background health checks * Added background healthcheck + half-open event * Refactored background scheduling * Added support for Active-Active pipeline * Refactored healthchecks * Added Pipeline testing * Added support for transactions * Removed code repetitions, fixed weight assignment, added loops enhancement, fixed data structure * Added missing doc blocks * Refactored configuration * Refactored failure detector * Refactored retry logic * Added scenario tests * Added pybreaker optional dependency * Added pybreaker to dev dependencies * Rename tests directory * Added scenario tests for Pipeline and Transaction * Added handling of ConnectionRefusedError, added timeouts so cluster could recover * Increased timeouts * Refactored integration tests * Fixed property name * Removed sentinels * Removed unused method

* Added Database, Healthcheck, CircuitBreaker, FailureDetector * Added DatabaseSelector, exceptions, refactored existing entities * Added MultiDbConfig * Added DatabaseConfig * Added DatabaseConfig test coverage * Renamed DatabaseSelector into FailoverStrategy * Added CommandExecutor * Updated healthcheck to close circuit on success * Added thread-safeness * Added missing thread-safeness * Added missing thread-safenes for dispatcher * Refactored client to keep databases in WeightedList * Added database CRUD operations * Added on-fly configuration * Added background health checks * Added background healthcheck + half-open event * Refactored background scheduling * Added support for Active-Active pipeline * Refactored healthchecks * Added Pipeline testing * Added support for transactions * Removed code repetitions, fixed weight assignment, added loops enhancement, fixed data structure * Added missing doc blocks * Added support for Pub/Sub in MultiDBClient * Refactored configuration * Refactored failure detector * Refactored retry logic * Added scenario tests * Added pybreaker optional dependency * Added pybreaker to dev dependencies * Rename tests directory * Added scenario tests for Pipeline and Transaction * Added handling of ConnectionRefusedError, added timeouts so cluster could recover * Increased timeouts * Refactored integration tests * Added scenario tests for Pub/Sub * Updated healthcheck retry * Increased timeout to avoid unprepared state before tests * Added backoff retry and changed timeouts * Added retry for healthchecks to avoid fluctuations * Changed retry configuration for healthchecks * Fixed property name * Added check for thread results

)

* Added Database, Healthcheck, CircuitBreaker, FailureDetector * Added DatabaseSelector, exceptions, refactored existing entities * Added MultiDbConfig * Added DatabaseConfig * Added DatabaseConfig test coverage * Renamed DatabaseSelector into FailoverStrategy * Added CommandExecutor * Updated healthcheck to close circuit on success * Added thread-safeness * Added missing thread-safeness * Added missing thread-safenes for dispatcher * Refactored client to keep databases in WeightedList * Added database CRUD operations * Added on-fly configuration * Added background health checks * Added background healthcheck + half-open event * Refactored background scheduling * Added support for Active-Active pipeline * Refactored healthchecks * Added Pipeline testing * Added support for transactions * Removed code repetitions, fixed weight assignment, added loops enhancement, fixed data structure * Added missing doc blocks * Added support for Pub/Sub in MultiDBClient * Refactored configuration * Refactored failure detector * Refactored retry logic * Added scenario tests * Added pybreaker optional dependency * Added pybreaker to dev dependencies * Rename tests directory * Added scenario tests for Pipeline and Transaction * Added handling of ConnectionRefusedError, added timeouts so cluster could recover * Increased timeouts * Refactored integration tests * Added scenario tests for Pub/Sub * Updated healthcheck retry * Increased timeout to avoid unprepared state before tests * Added backoff retry and changed timeouts * Added retry for healthchecks to avoid fluctuations * Changed retry configuration for healthchecks * Fixed property name * Added check for thread results * Added MultiDbClient support with OSS Cluster API * Removed database statuses * Increased test timeouts * Increased retry timeout * Increased timeout retries * Updated base threshold for retries * Fixed flacky tests * Added missing docblocks

* Added LagAwareHealthcheck * Added testing for LagAwareHealthCheck * Fixed timeouts * Added lag tollerance parameter * Decreased messages_count due to increased timeouts * Added docblocks * Added missing type hints * Fixed url * Refactored tests, URL and cluster support * Use primary node to send an API request * Added comment about RE bug * Moved None type to the beginning * Added health_check_url property to Database class

* Extract additional interfaces and abstract classes * Added base async components * Added command executor * Added recurring background tasks with event loop only * Added MultiDBClient * Added scenario and config tests * Update redis/asyncio/multidb/healthcheck.py Co-authored-by: Copilot <[email protected]> * Update tests/test_asyncio/test_scenario/test_active_active.py Co-authored-by: Copilot <[email protected]> --------- Co-authored-by: Copilot <[email protected]>

* Extract additional interfaces and abstract classes * Added base async components * Added command executor * Added recurring background tasks with event loop only * Added MultiDBClient * Added scenario and config tests * Added pipeline and transaction support for MultiDBClient * Updated scenario tests to check failover

* Extract additional interfaces and abstract classes * Added base async components * Added command executor * Added recurring background tasks with event loop only * Added MultiDBClient * Added scenario and config tests * Added pipeline and transaction support for MultiDBClient * Added pub/sub support for MultiDBClient * Added check for couroutines methods for pub/sub

* Extract additional interfaces and abstract classes * Added base async components * Added command executor * Added recurring background tasks with event loop only * Added MultiDBClient * Added scenario and config tests * Added pipeline and transaction support for MultiDBClient * Added pub/sub support for MultiDBClient * Added check for couroutines methods for pub/sub * Added OSS Cluster API support for MultiDBCLient * Added support for Lag-Aware Healthcheck and OSS Cluster API * Increased timeouts between tests * Fixed space

* Extract additional interfaces and abstract classes * Added base async components * Added command executor * Added recurring background tasks with event loop only * Added MultiDBClient * Added scenario and config tests * Added pipeline and transaction support for MultiDBClient * Added pub/sub support for MultiDBClient * Added check for couroutines methods for pub/sub * Added OSS Cluster API support for MultiDBCLient * Added support for Lag-Aware Healthcheck and OSS Cluster API * Increased timeouts between tests * [Sync] Refactored healthcheck * [Async] Refactored healthcheck * [Sync] Refactored Failover Strategy * [Async] Refactored Failover Strategy * Changed default values according to a design doc * [Async] Added Strategy Executor * [Sync] Added Strategy Executor * Apply comments

…ctive

…dling

…ctive

jit-ci · 2025-10-02T07:50:59Z

Hi, I’m Jit, a friendly security platform designed to help developers build secure applications from day zero with an MVS (Minimal viable security) mindset.

In case there are security findings, they will be communicated to you as a comment inside the PR.

Hope you’ll enjoy using Jit.

Questions? Comments? Want to learn more? Get in touch with us.

…eat/active-active

vladvildanov and others added 30 commits August 11, 2025 11:12

Refactored docblocks (#3744)

fb500f6

Refactored healthcheck and failure detector to extend default one (#3747

d6cdaeb

)

Added lag_aware_tolerance parameter to LagAwareHealthcheck (#3752)

f9fdc99

Extract additional interfaces and abstract classes (#3754)

866003b

Merge branch 'master' of github.com:redis/redis-py into feat/active-a…

5dff45a

…ctive

Removed redundant dependency

457a35c

Fixed async tests

c21da4b

Increased lag-aware tolerance

821cc54

Fixed typing issue, increase health_check_interval, added timeout han…

a7540c1

…dling

Decreased retry cap, increased failure delay

ca8166c

Fixed async teardown

f50299e

Fixed tests

115d996

Added graceful connection closing, added graceful hc tasks termination

6556e8a

Make sure active connection will be disconnected on failover

063e795

Close cluster connection on failover

3ef34b1

Refactored Failure Detector (#3775)

a1c0633

Decreased timeouts

f5231ee

Added missing fixture

413ea86

Fixed None exception

3ed14e4

Merge branch 'master' of github.com:redis/redis-py into feat/active-a…

6b0c4ad

…ctive

vladvildanov requested a review from petyaslavova October 2, 2025 07:50

vladvildanov added 6 commits October 2, 2025 10:52

Codestyle changes

71fc90f

Codestyle changes

57d328b

Skip async scenario tests

18b5b52

Codestyle change

2ad005a

Fixed unused arguments

2672fce

Refactored bg scheduler

be5d2e8

vladvildanov added the feature New feature label Oct 2, 2025

vladvildanov and others added 13 commits October 2, 2025 14:54

Fixed tests

bca7a31

Fixed tests

d9ad720

Codestyle fixes

031b705

Reduce timeouts to avoid overlaping with healthcheck

e3bfa78

Marked tests non-clsuter only

5d5ff26

Update timeouts

cb51f8a

Merge branch 'master' into feat/active-active

d87ab3d

Skip scenario tests

7f7ea76

Merge branch 'feat/active-active' of github.com:redis/redis-py into f…

852e835

…eat/active-active

Updated timeouts

933b3ad

Merge branch 'master' into feat/active-active

c350d61

Increased timeout

cdef58b

Merge branch 'feat/active-active' of github.com:redis/redis-py into f…

003853b

…eat/active-active

petyaslavova approved these changes Oct 3, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Multi-database client implementation #3784

Multi-database client implementation #3784

Uh oh!

vladvildanov commented Oct 2, 2025

Uh oh!

jit-ci bot commented Oct 2, 2025

Uh oh!

Uh oh!

Multi-database client implementation #3784

Are you sure you want to change the base?

Multi-database client implementation #3784

Uh oh!

Conversation

vladvildanov commented Oct 2, 2025

Pull Request check-list

Description of change

Uh oh!

jit-ci bot commented Oct 2, 2025

Uh oh!

Uh oh!