Mapping keys are never deleted with Redis

[ungive]

I'm using the rueidis (redis) storage backend for caching. Instead of a plain Redis instance I'm using Pika for persistent/disk storage (https://github.com/OpenAtomFoundation/pika), but I think the same should apply to normal redis instances, since Pika is redis compatible and I've confirmed the behaviour I'm about to describe by monitoring incoming commands (using MONITOR in redis CLI).

I have noticed that my database keeps growing indefinitely over time and never shrinks:

Using redis CLI I found out that there are tons of keys prefixed with IDX_ with no TTL (-1). Initially I thought those were the direct caching keys and I wondered why my TTL from the config wasn't applied, but looking closer and reading the code in this repo I realised it's the mapping keys that keep getting added but are never deleted. IDX_ is the mapping key prefix and it appears to never be removed in redis.go:

storages/core/core.go

Line 44 in 88bb38d

const MappingKeyPrefix = "IDX_"

https://github.com/darkweak/storages/blob/88bb38dfa2b666d2473086577978782018bc6638/redis/redis.go

I think they should be deleted at some point, when they're no longer useful, no? I'm not that familiar with Redis, maybe normally they get cleaned up automatically somehow and the maintainer did not think that deleting them or giving them an expire value would be necessary, but in my case it's absolutely needed to delete them when they are no longer required.

This also causes the CPU usage of Caddy and my database to slowly increase over time (Pika and Caddy respectively):

I assume that's cause of the scanning done here:

storages/redis/redis.go

Line 108 in 88bb38d

if scan, err = provider.inClient.Do(context.Background(), provider.inClient.B().Scan().Cursor(scan.Cursor).Match(provider.hashtags+core.MappingKeyPrefix+"*").Build()).AsScanEntry(); err != nil {

Any input on this is welcome, thank you!

[xzycn]

@darkweak
Hi, sorry to bother you, I have checked the code related to this issue in darkweak/souin@1a941dc .
However, I don't see the action to purge the expired keys (prefix with IDX_), but scan and get every minute, did I miss something? Thx.

[xzycn]

@darkweak
Hi, can we check this?

[g-ra]

@xzycn
problem in SURROGATE_KEYS
use my PR for tests please
darkweak/souin#645
Example Dockerfile

   --with github.com/darkweak/souin \
    --with github.com/darkweak/souin/plugins/caddy \
    --with github.com/darkweak/storages/go-redis/caddy \
    # --- my forks ---
    --replace github.com/darkweak/souin=github.com/g-ra/souin@5f639fe758a122018ee8cdaa17c16e955a61bcbc