好吧,我发现 hash_ring.MemcacheRing 太慢了,rate test 无法直接击败 db access 而,我用 memcache.Client 替换它,rate test 恢复正常 rps 是在多线程环境下测试的
memcache.Client rps2500
hash_ring.MemcacheRing rps 600
directly db access:rps960
我使用hotshot来跟踪hash_ring.MemcacheRing,我发现hash_ring肯定有问题,你可以发现hash_ring花费太多的cpu,在以下配置文件数据收集的热点配置文件数据中收集在单线程环境中
ncalls tottime percall cumtime percall filename:lineno(function) 10000 0.066 0.000 8.297 0.001 Memcached.py:16(__getitem__) 10000 0.262 0.000 6.544 0.001 构建/bdist.linux-x86_64/egg/memcache.py:818(_unsafe_get) 1 0.000 0.000 0.000 0.000 /usr/local/Python-2.6.4/lib/python2.6/socket.py:180(__init__) 20000 0.064 0.000 0.064 0.000 build/bdist.linux-x86_64/egg/memcache.py:310(_statlog) 80000 0.118 0.000 0.118 0.000 build/bdist.linux-x86_64/egg/hash_ring/hash_ring.py:148() 10000 0.148 0.000 8.168 0.001 构建/bdist.linux-x86_64/egg/memcache.py:812(_get) 20000 0.086 0.000 0.157 0.000 构建/bdist.linux-x86_64/egg/memcache.py:1086(_get_socket) 30000 10.539 0.000 10.539 0.000 build/bdist.linux-x86_64/egg/memcache.py:1118(readline) 10000 0.102 0.000 0.102 0.000 build/bdist.linux-x86_64/egg/memcache.py:1142(recv) 20000 0.243 0.000 0.243 0.000 构建/bdist.linux-x86_64/egg/hash_ring/hash_ring.py:156(_hash_digest) 20000 0.103 0.000 0.103 0.000 build/bdist.linux-x86_64/egg/hash_ring/hash_ring.py:124(distinct_filter) 10000 0.167 0.000 0.269 0.000 构建/bdist.linux-x86_64/egg/memcache.py:965(_recv_value) 1 0.148 0.148 15.701 15.701 Memcached.py:25(foo) 10000 0.236 0.000 5.409 0.001 构建/bdist.linux-x86_64/egg/memcache.py:953(_expectvalue) 20002 0.336 0.000 0.336 0.000 :1(设置超时) 10000 0.061 0.000 7.207 0.001 构建/bdist.linux-x86_64/egg/memcache.py:541(设置) 20000 0.116 0.000 0.606 0.000 build/bdist.linux-x86_64/egg/hash_ring/hash_ring.py:141(gen_key) 20000 0.279 0.000 1.680 0.000 build/bdist.linux-x86_64/egg/hash_ring/memcache_ring.py:20(_get_server) 40000 0.346 0.000 1.236 0.000 构建/bdist.linux-x86_64/egg/hash_ring/hash_ring.py:111(迭代节点) 10000 0.144 0.000 7.146 0.001 构建/bdist.linux-x86_64/egg/memcache.py:771(_set) 20000 0.129 0.000 0.247 0.000 构建/bdist.linux-x86_64/egg/hash_ring/hash_ring.py:150(_hash_val) 20000 0.098 0.000 0.433 0.000 build/bdist.linux-x86_64/egg/memcache.py:1111(send_cmd) 10000 0.063 0.000 8.231 0.001 build/bdist.linux-x86_64/egg/memcache.py:857(get) 10000 0.049 0.000 7.256 0.001 Memcached.py:13(__setitem__) 20000 0.144 0.000 5.510 0.000 build/bdist.linux-x86_64/egg/memcache.py:1135(预期) 20000 0.181 0.000 0.787 0.000 构建/bdist.linux-x86_64/egg/hash_ring/hash_ring.py:92(get_node_pos) 20000 0.070 0.000 0.070 0.000 build/bdist.linux-x86_64/egg/memcache.py:1070(_check_dead) 10000 0.084 0.000 0.084 0.000 构建/bdist.linux-x86_64/egg/memcache.py:725(_val_to_store_info) 20000 0.068 0.000 0.225 0.000 build/bdist.linux-x86_64/egg/memcache.py:1076(连接) 20000 1.036 0.000 1.036 0.000 build/bdist.linux-x86_64/egg/memcache.py:1000(check_key) 10000 0.216 0.000 5.702 0.001 构建/bdist.linux-x86_64/egg/memcache.py:777(_unsafe_set) 0 0.000 0.000 轮廓:0(轮廓仪)
以下是 memcache.Client 的配置文件数据
ncalls tottime percall cumtime percall filename:lineno(function) 10000 0.064 0.000 8.022 0.001 Memcached.py:16(__getitem__) 10000 0.263 0.000 6.885 0.001 构建/bdist.linux-x86_64/egg/memcache.py:818(_unsafe_get) 1 0.000 0.000 0.000 0.000 /usr/local/Python-2.6.4/lib/python2.6/socket.py:180(__init__) 20000 0.069 0.000 0.069 0.000 build/bdist.linux-x86_64/egg/memcache.py:310(_statlog) 10000 0.127 0.000 7.897 0.001 构建/bdist.linux-x86_64/egg/memcache.py:812(_get) 20000 0.091 0.000 0.164 0.000 构建/bdist.linux-x86_64/egg/memcache.py:1086(_get_socket) 30000 11.074 0.000 11.074 0.000 build/bdist.linux-x86_64/egg/memcache.py:1118(readline) 10000 0.098 0.000 0.098 0.000 构建/bdist.linux-x86_64/egg/memcache.py:1142(recv) 20000 0.259 0.000 0.568 0.000 build/bdist.linux-x86_64/egg/memcache.py:329(_get_server) 1 0.149 0.149 15.036 15.036 Memcached.py:25(foo) 10000 0.236 0.000 5.645 0.001 build/bdist.linux-x86_64/egg/memcache.py:953(_expectvalue) 20002 0.351 0.000 0.351 0.000 :1(设置超时) 10000 0.064 0.000 6.814 0.001 构建/bdist.linux-x86_64/egg/memcache.py:541(设置) 10000 0.124 0.000 6.751 0.001 构建/bdist.linux-x86_64/egg/memcache.py:771(_set) 20000 1.039 0.000 1.039 0.000 build/bdist.linux-x86_64/egg/memcache.py:1000(check_key) 20000 0.092 0.000 0.442 0.000 build/bdist.linux-x86_64/egg/memcache.py:1111(send_cmd) 10000 0.061 0.000 7.958 0.001 构建/bdist.linux-x86_64/egg/memcache.py:857(获取) 10000 0.050 0.000 6.864 0.001 Memcached.py:13(__setitem__) 20000 0.148 0.000 5.812 0.000 build/bdist.linux-x86_64/egg/memcache.py:1135(预期) 20000 0.073 0.000 0.073 0.000 build/bdist.linux-x86_64/egg/memcache.py:1070(_check_dead) 10000 0.087 0.000 0.087 0.000 build/bdist.linux-x86_64/egg/memcache.py:725(_val_to_store_info) 20000 0.072 0.000 0.236 0.000 build/bdist.linux-x86_64/egg/memcache.py:1076(连接) 20000 0.072 0.000 0.072 0.000 构建/bdist.linux-x86_64/egg/memcache.py:57(cmemcache_hash) 10000 0.218 0.000 5.905 0.001 构建/bdist.linux-x86_64/egg/memcache.py:777(_unsafe_set) 0 0.000 0.000 轮廓:0(轮廓仪) 10000 0.156 0.000 0.255 0.000 build/bdist.linux-x86_64/egg/memcache.py:965(_recv_value)
和个人资料数据如下
ncalls tottime percall cumtime percall filename:lineno(function) 20000 2.278 0.000 4.260 0.000 build/bdist.linux-x86_64/egg/memcache.py:1000(check_key) 640000 1.624 0.000 1.624 0.000 :0 (ord) 20000 0.658 0.000 1.445 0.000 构建/bdist.linux-x86_64/egg/memcache.py:329(_get_server) 100010 0.477 0.000 0.477 0.000 :0(len) 30000 0.460 0.000 0.827 0.000 build/bdist.linux-x86_64/egg/memcache.py:1118(readline) 110000 0.414 0.000 0.414 0.000 :0(实例) 20000 0.412 0.000 0.412 0.000 :0(发送) 10000 0.349 0.000 1.974 0.000 构建/bdist.linux-x86_64/egg/memcache.py:818(_unsafe_get) 10000 0.344 0.000 0.633 0.000 构建/bdist.linux-x86_64/egg/memcache.py:725(_val_to_store_info) 10000 0.344 0.000 2.006 0.000 build/bdist.linux-x86_64/egg/memcache.py:777(_unsafe_set) 10000 0.325 0.000 0.624 0.000 构建/bdist.linux-x86_64/egg/memcache.py:965(_recv_value) 50000 0.233 0.000 0.233 0.000 :0(查找) 20000 0.200 0.000 0.808 0.000 build/bdist.linux-x86_64/egg/memcache.py:1135(预期) 10000 0.194 0.000 5.292 0.001 构建/bdist.linux-x86_64/egg/memcache.py:812(_get) 20000 0.190 0.000 0.357 0.000 build/bdist.linux-x86_64/egg/memcache.py:1076(连接) 10000 0.185 0.000 4.771 0.000 构建/bdist.linux-x86_64/egg/memcache.py:771(_set) 10000 0.173 0.000 0.187 0.000 构建/bdist.linux-x86_64/egg/memcache.py:1142(recv) 10000 0.163 0.000 0.400 0.000 build/bdist.linux-x86_64/egg/memcache.py:953(_expectvalue) 20000 0.152 0.000 0.564 0.000 :1(发送) 20000 0.147 0.000 0.711 0.000 build/bdist.linux-x86_64/egg/memcache.py:1111(send_cmd) 20000 0.142 0.000 0.232 0.000 构建/bdist.linux-x86_64/egg/memcache.py:57(cmemcache_hash)
ncalls tottime percall cumtime percall filename:lineno(function) 20000 2.679 0.000 4.921 0.000 build/bdist.linux-x86_64/egg/memcache.py:1000(check_key) 640000 2.125 0.000 2.125 0.000 :0 (ord) 20000 0.636 0.000 2.887 0.000 构建/bdist.linux-x86_64/egg/hash_ring/memcache_ring.py:20(_get_server) 20000 0.582 0.000 1.673 0.000 构建/bdist.linux-x86_64/egg/hash_ring/hash_ring.py:92(get_node_pos) 30000 0.565 0.000 0.979 0.000 build/bdist.linux-x86_64/egg/memcache.py:1118(readline) 20000 0.345 0.000 0.630 0.000 构建/bdist.linux-x86_64/egg/hash_ring/hash_ring.py:156(_hash_digest) 40000 0.328 0.000 2.065 0.000 构建/bdist.linux-x86_64/egg/hash_ring/hash_ring.py:111(迭代节点) 20003 0.294 0.000 0.294 0.000 :0(接收) 10000 0.287 0.000 1.119 0.000 build/bdist.linux-x86_64/egg/memcache.py:953(_expectvalue) 10000 0.176 0.000 4.437 0.000 构建/bdist.linux-x86_64/egg/memcache.py:771(_set) 10000 0.175 0.000 1.697 0.000 构建/bdist.linux-x86_64/egg/memcache.py:818(_unsafe_get) 10000 0.160 0.000 5.899 0.001 构建/bdist.linux-x86_64/egg/memcache.py:812(_get) 20000 0.157 0.000 1.051 0.000 build/bdist.linux-x86_64/egg/hash_ring/hash_ring.py:141(gen_key) 10000 0.148 0.000 0.251 0.000 构建/bdist.linux-x86_64/egg/memcache.py:965(_recv_value) 20000 0.144 0.000 0.264 0.000 构建/bdist.linux-x86_64/egg/hash_ring/hash_ring.py:150(_hash_val) 90000 0.122 0.000 0.122 0.000 :0(实例) 50000 0.120 0.000 0.120 0.000 :0(查找) 80000 0.120 0.000 0.120 0.000 构建/bdist.linux-x86_64/egg/hash_ring/hash_ring.py:148()