-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Description
I noticed this issue when using Spring Boot Webflux (v3.4.10), Spring Framework (v6.2.11) and Spring Data Redis (v3.4.10).
Our service is using a Redis cache with support provided by Lettuce (v6.4.2.RELEASE).
The service uses the @Cacheable
and @CacheEvict
annotations with reactive support:
@Cacheable(RedisCacheConfig.IDs.ATTRIBUTES_CACHE)
fun getAttributes(id: Identifier): Mono<List<Attribute>> =
attributesService.getAttributesById(id)
@CacheEvict(RedisCacheConfig.IDs.ATTRIBUTES_CACHE)
fun evictAttributes(id: Identifier): Mono<Unit> = Mono.empty()
Calling code was evicting some keys if they happened to return an empty list:
idList.associateWith { id ->
attributesCache.getAttributes(id)
.flatMap { attributes ->
// evict cached attributes from redis if the attributes list is empty
if (attributes.isEmpty()) {
attributesCache
.evictRelevantAssetAttributes(id)
.then(Mono.just(attributes))
} else {
Mono.just(attributes)
}
}
.awaitSingle()
}
We underestimated the number of times that we would be evicting cache entries and the service eventually became unresponsive and started logging errors due to 1 minute cache command timeouts (for get, store & evictions).
Digging into the problem we noticed that the number of blocking JVM threads spiked during this time. Using a debugger it is possible to trace the eviction flow and it looks like there is a blocking operation going on during cache eviction.
The Spring Framework is interacting through Spring Data Redis via the Cache interface which notes in the JavaDocs for the evict method:
If the cache is supposed to be compatible with CompletableFuture and reactive interactions, the evict operation needs to be effectively non-blocking, with any backend write-through happening asynchronously.
It looks like using Spring Data Redis with Lettuce supports async/reactive retrieve & store operations via the AsynchronousCacheWriterDelegate
class.
However, it also looks like calls to evict a key use the synchronous cache writer via the DefaultRedisCacheWriter which blocks when invoking the del command via LettuceKeyCommands
in LettuceConnection.
Is there a version / configuration of Spring Data Redis that supports async cache evictions? or is that functionality missing?