51

I'm writing a django management command to handle some of our redis caching. Basically, I need to choose all keys, that confirm to a certain pattern (for example: "prefix:*") and delete them.

I know I can use the cli to do that:

redis-cli KEYS "prefix:*" | xargs redis-cli DEL

But I need to do this from within the app. So I need to use the python binding (I'm using py-redis). I have tried feeding a list into delete, but it fails:

from common.redis_client import get_redis_client
cache = get_redis_client()
x = cache.keys('prefix:*') 

x == ['prefix:key1','prefix:key2'] # True

# And now

cache.delete(x)

# returns 0 . nothing is deleted

I know I can iterate over x:

for key in x:
   cache.delete(key)

But that would be losing redis awesome speed and misusing its capabilities. Is there a pythonic solution with py-redis, without iteration and/or the cli?

Thanks!

4

9 回答 9

49

使用 SCAN 迭代器：https ://pypi.python.org/pypi/redis

for key in r.scan_iter("prefix:*"):
    r.delete(key)
于 2016-05-10T09:59:28.190 回答
28

这是使用py-redis的完整工作示例：

from redis import StrictRedis
cache = StrictRedis()

def clear_ns(ns):
    """
    Clears a namespace
    :param ns: str, namespace i.e your:prefix
    :return: int, cleared keys
    """
    count = 0
    ns_keys = ns + '*'
    for key in cache.scan_iter(ns_keys):
        cache.delete(key)
        count += 1
    return count

您也可以scan_iter将所有键放入内存，然后将所有键传递delete给批量删除，但对于较大的命名空间可能会占用大量内存。所以可能最好delete为每个键运行一个。

干杯!

更新：

写完答案后，我开始使用 redis 的流水线功能在一个请求中发送所有命令并避免网络延迟：

from redis import StrictRedis
cache = StrictRedis()

def clear_cache_ns(ns):
    """
    Clears a namespace in redis cache.
    This may be very time consuming.
    :param ns: str, namespace i.e your:prefix*
    :return: int, num cleared keys
    """
    count = 0
    pipe = cache.pipeline()
    for key in cache.scan_iter(ns):
        pipe.delete(key)
        count += 1
    pipe.execute()
    return count

UPDATE2（表现最佳）：

如果使用scan代替scan_iter，则可以控制块大小并使用自己的逻辑遍历游标。这似乎也快了很多，尤其是在处理许多键时。如果为此添加流水线，您将获得一点性能提升，10-25% 取决于块大小，但会以内存使用为代价，因为在生成所有内容之前您不会将执行命令发送到 Redis。所以我坚持扫描：

from redis import StrictRedis
cache = StrictRedis()
CHUNK_SIZE = 5000

def clear_ns(ns):
    """
    Clears a namespace
    :param ns: str, namespace i.e your:prefix
    :return: int, cleared keys
    """
    cursor = '0'
    ns_keys = ns + '*'
    while cursor != 0:
        cursor, keys = cache.scan(cursor=cursor, match=ns_keys, count=CHUNK_SIZE)
        if keys:
            cache.delete(*keys)

    return True

以下是一些基准：

使用繁忙的 Redis 集群的 5k 块：

Done removing using scan in 4.49929285049
Done removing using scan_iter in 98.4856731892
Done removing using scan_iter & pipe in 66.8833789825
Done removing using scan & pipe in 3.20298910141

5k 块和一个小的空闲开发 redis（本地主机）：

Done removing using scan in 1.26654982567
Done removing using scan_iter in 13.5976779461
Done removing using scan_iter & pipe in 4.66061878204
Done removing using scan & pipe in 1.13942599297
于 2017-07-17T20:55:51.090 回答
25

我觉得

 for key in x: cache.delete(key)

相当不错，简洁。 delete一次真的想要一个键，所以你必须循环。

否则，前面的问题和答案将指向基于 lua 的解决方案。

于 2014-02-23T22:17:48.470 回答
10

从文档

delete(*names)
    Delete one or more keys specified by names

这只是想要删除每个键的参数，然后它会告诉您找到并删除了其中的多少。

对于上面的代码，我相信您可以这样做：

    redis.delete(*x)

但我承认我是 python 新手，我只是这样做：

    deleted_count = redis.delete('key1', 'key2')
于 2016-05-17T02:34:40.490 回答
5

cache.delete(*keys)Dirk 的解决方案工作正常，但确保键不为空以避免redis.exceptions.ResponseError: wrong number of arguments for 'del' command.

如果您确定总会得到结果：cache.delete(*cache.keys('prefix:*') )

于 2014-10-31T15:03:56.367 回答
3

顺便说一句，对于 django-redis，您可以使用以下内容（来自https://niwinz.github.io/django-redis/latest/）：

from django.core.cache import cache
cache.delete_pattern("foo_*")
于 2016-02-25T14:15:54.917 回答
3

根据我的测试，如果我使用解决方案将花费太多时间scan_iter（正如Alex Toderita 所写）。

因此，我更喜欢使用：

from redis.connection import ResponseError

try:
    redis_obj.eval('''return redis.call('del', unpack(redis.call('keys', ARGV[1])))''', 0, 'prefix:*')
except ResponseError:
    pass

是prefix:*模式。

指： https ://stackoverflow.com/a/16974060

于 2017-11-07T02:35:01.270 回答
2

您可以使用特定模式来匹配所有键并删除它们：

import redis
client = redis.Redis(host='192.168.1.106', port=6379,
                password='pass', decode_responses=True)
for key in client.keys('prefix:*'):
    client.delete(key)
于 2018-09-12T09:55:38.403 回答
1

使用 delete_pattern：https ://niwinz.github.io/django-redis/latest/

from django.core.cache import cache
cache.delete_pattern("prefix:*")
于 2017-12-15T04:41:53.160 回答