0

这是我第一次在这里发布问题，所以请忽略与标准格式的偏差...

这是我的设置。Nginx --> Uwsgi --> Django --> (Memcached/Mysql)

我在 Django 中有一组返回 json 响应的 API。早些时候，我使用 memcached 2 部分中间件进行自动缓存。现在为了获得更多控制权，我已经转向低级 API。我使用 celery 创建腌制数据集并每天一次在 memcached 中插入数据，My APIs 从 memcached 中获取数据。

我在我的生产环境中使用 NGINX 和 Uwsgi（16 个工作人员），并且通过 2 部分中间件方法获得了大约 8K 请求的速度。只有我的第一个请求过去很慢。但是使用较低级别的缓存，我得到了 1/10 的速度。

最初我认为这是由于额外的中间件和正在执行的自定义代码，但我也对其进行了调整。我从 pickle 搬到 cPickle 但我不认为这是瓶颈。我正在使用 apache benchmark 进行测试。

我正在为中间件方法和低级缓存 api 附加 Memcached (-vv) 日志。如果有人能指出我正确的方向............谢谢

我看到了 Django 的 catche-middleware 代码。他们似乎将 django httpresponse 对象保存在缓存中。中间件也在内部调用低级 api。我也尝试将 httpresponse 保存在缓存中，但没有改变......

日志

<pre>

MIDDLEWARE approach memcached LOGS
<28 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>28 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>28 END
<28 connection closed.
<28 new auto-negotiating client connection
28: Client using the ascii protocol
<28 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>28 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>28 END
<29 new auto-negotiating client connection
29: Client using the ascii protocol
<29 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>29 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>29 END
<30 new auto-negotiating client connection
30: Client using the ascii protocol
<30 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>30 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>30 END
<32 new auto-negotiating client connection
32: Client using the ascii protocol
<32 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>32 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>32 END
<31 new auto-negotiating client connection
31: Client using the ascii protocol
<31 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>31 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>31 END
<33 new auto-negotiating client connection
33: Client using the ascii protocol
<33 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>33 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>33 END
<34 new auto-negotiating client connection
34: Client using the ascii protocol
<34 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>34 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>34 END
<35 new auto-negotiating client connection
35: Client using the ascii protocol
<35 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>35 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>35 END
<33 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>33 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>33 END
<28 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>28 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>28 END
<37 new auto-negotiating client connection
37: Client using the ascii protocol
<37 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>37 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
<36 new auto-negotiating client connection
36: Client using the ascii protocol
<36 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>36 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>36 END
<40 new auto-negotiating client connection
40: Client using the ascii protocol
<40 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>40 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>40 END
<30 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>30 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>30 END
<34 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>34 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>34 END
<38 new auto-negotiating client connection
<42 new auto-negotiating client connection
38: Client using the ascii protocol
<38 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>38 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>38 END
42: Client using the ascii protocol
<42 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>42 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>42 END
>37 END
<41 new auto-negotiating client connection
<33 connection closed.
<36 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
<31 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>36 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>36 END
41: Client using the ascii protocol
<41 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>41 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>41 END
>31 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>31 END
<39 new auto-negotiating client connection
39: Client using the ascii protocol
<39 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>39 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>39 END
<38 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>38 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>38 END
<33 new auto-negotiating client connection
33: Client using the ascii protocol
<33 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>33 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>33 END
<31 connection closed.
<36 connection closed.
<29 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>29 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>29 END
<33 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>33 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>33 END
<39 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>39 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>39 END
<32 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>32 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>32 END
<35 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>35 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>35 END
<33 connection closed.
<30 connection closed.
<42 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>42 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>42 END
<37 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>37 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>37 END
<35 connection closed.
<39 connection closed.
<32 connection closed.
<31 new auto-negotiating client connection
31: Client using the ascii protocol
<31 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
<30 new auto-negotiating client connection
>31 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
30: Client using the ascii protocol
<30 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>31 END
>30 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>30 END
<32 new auto-negotiating client connection
32: Client using the ascii protocol
<32 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>32 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>32 END
<32 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>32 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>32 END
<30 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>30 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>30 END
<31 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>31 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>31 END
<33 new auto-negotiating client connection
33: Client using the ascii protocol
<33 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>33 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>33 END
<33 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>33 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>33 END
<30 connection closed.
<30 new auto-negotiating client connection
30: Client using the ascii protocol
<30 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>30 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>30 END
<38 connection closed.
<32 connection closed.
<30 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>30 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>30 END
<33 connection closed.
<37 connection closed.
<31 connection closed.
<29 connection closed.
<29 new auto-negotiating client connection
29: Client using the ascii protocol
<29 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>29 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>29 END
<34 connection closed.
<42 connection closed.
<41 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>41 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>41 END
<40 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>40 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>40 END
<28 connection closed.
<30 connection closed.
<29 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>29 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>29 END
<28 new auto-negotiating client connection
28: Client using the ascii protocol
<28 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>28 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>28 END
<28 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>28 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>28 END
<30 new auto-negotiating client connection
<32 new auto-negotiating client connection
32: Client using the ascii protocol
<32 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>32 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>32 END
<40 connection closed.

</pre>


----------


----------


LOW level logs
<pre>
<43 connection closed.
<43 new auto-negotiating client connection
43: Client using the ascii protocol
<43 get :1:ls|1|0|10|||
>43 sending key :1:ls|1|0|10|||
>43 END
<33 connection closed.
<33 new auto-negotiating client connection
33: Client using the ascii protocol
<33 get :1:ls|1|0|10|||
>33 sending key :1:ls|1|0|10|||
>33 END
<32 connection closed.
<32 new auto-negotiating client connection
32: Client using the ascii protocol
<32 get :1:ls|1|0|10|||
>32 sending key :1:ls|1|0|10|||
>32 END
<36 connection closed.
<36 new auto-negotiating client connection
36: Client using the ascii protocol
<36 get :1:ls|1|0|10|||
>36 sending key :1:ls|1|0|10|||
>36 END
<40 connection closed.
<40 new auto-negotiating client connection
40: Client using the ascii protocol
<40 get :1:ls|1|0|10|||
>40 sending key :1:ls|1|0|10|||
>40 END
<34 connection closed.
<34 new auto-negotiating client connection
34: Client using the ascii protocol
<34 get :1:ls|1|0|10|||
>34 sending key :1:ls|1|0|10|||
>34 END
<29 connection closed.
<29 new auto-negotiating client connection
29: Client using the ascii protocol
<29 get :1:ls|1|0|10|||
>29 sending key :1:ls|1|0|10|||
>29 END
<35 connection closed.
<35 new auto-negotiating client connection
35: Client using the ascii protocol
<35 get :1:ls|1|0|10|||
>35 sending key :1:ls|1|0|10|||
>35 END
<37 connection closed.
<37 new auto-negotiating client connection
37: Client using the ascii protocol
<37 get :1:ls|1|0|10|||
>37 sending key :1:ls|1|0|10|||
>37 END
<41 connection closed.
<41 new auto-negotiating client connection
41: Client using the ascii protocol
<41 get :1:ls|1|0|10|||
>41 sending key :1:ls|1|0|10|||
>41 END
<39 connection closed.
<39 new auto-negotiating client connection
39: Client using the ascii protocol
<39 get :1:ls|1|0|10|||
>39 sending key :1:ls|1|0|10|||
>39 END
<38 connection closed.
<38 new auto-negotiating client connection
38: Client using the ascii protocol
<38 get :1:ls|1|0|10|||
>38 sending key :1:ls|1|0|10|||
>38 END
<30 connection closed.
<30 new auto-negotiating client connection
30: Client using the ascii protocol
<30 get :1:ls|1|0|10|||
>30 sending key :1:ls|1|0|10|||
>30 END
<31 connection closed.
<31 new auto-negotiating client connection
31: Client using the ascii protocol
<31 get :1:ls|1|0|10|||
>31 sending key :1:ls|1|0|10|||
>31 END
<42 connection closed.
<42 new auto-negotiating client connection
42: Client using the ascii protocol
<42 get :1:ls|1|0|10|||
>42 sending key :1:ls|1|0|10|||
>42 END
<28 connection closed.
<28 new auto-negotiating client connection
28: Client using the ascii protocol
<28 get :1:ls|1|0|10|||
>28 sending key :1:ls|1|0|10|||
>28 END
<43 connection closed.
<43 new auto-negotiating client connection
43: Client using the ascii protocol
<43 get :1:ls|1|0|10|||
>43 sending key :1:ls|1|0|10|||
>43 END
<33 connection closed.
<33 new auto-negotiating client connection
33: Client using the ascii protocol
<33 get :1:ls|1|0|10|||
>33 sending key :1:ls|1|0|10|||
>33 END
<32 connection closed.
<34 connection closed.
<32 new auto-negotiating client connection
32: Client using the ascii protocol
<32 get :1:ls|1|0|10|||
>32 sending key :1:ls|1|0|10|||

</pre>
4

1 回答 1

1

经过一周的尝试和测试（感谢我的团队），我意识到 Django 低级 API 有利于在数据库获取方面取得优势，但对于严重的缓存性能却没有。Django 中间件类仅在中间件堆栈吞吐量服务器进程中实例化一次。正如Django 官方文档和django book中提到的。

在使用低级 api 时，与 Memcache 的连接启动可能花费了太多时间。最后，我继承了FetchfromCache 中间件并创建了自己的自定义中间件，该中间件的性能与 Django 的两部分中间件相同。剥离不需要的中间件也确实有助于提高性能。

于 2013-07-16T14:02:05.410 回答