14

我一直在测试我制作的缓存系统。其目的是加速 Django Web 应用程序。它将所有内容存储在内存中。根据 cProfile，我的测试中的大部分时间都花在 QuerySet._clone() 中，结果证明效率非常低（考虑到实现，这实际上并不奇怪）。

我对使用 PyPy 加快速度寄予厚望。我有一台 64 位的机器。然而，在安装了所有必需的库之后，结果证明 PyPy 编译代码的运行速度比常规 Python 代码慢 2.5 倍，而且我不知道该怎么做。该代码受 CPU 限制（绝对没有数据库查询，因此不能选择 IO 限制）。一个测试运行大约 10 秒，所以我想 JIT 应该足够了。我使用的是 PyPy 1.5。注意——我自己没有编译源代码，只是下载了一个 64 位的 linux 版本。

我想知道 CPU 密集型代码在 PyPy 下实际运行速度变慢的频率。希望我做错了什么会阻止 PyPy 以最佳状态运行。

编辑

确切的 cPython 输出：

PyPy 1.5：

    3439146 function calls (3218654 primitive calls) in 19.094 seconds

    Ordered by: cumulative time

    ncalls  tottime  percall  cumtime  percall filename:lineno(function)
       2/1    0.000    0.000   18.956   18.956 <string>:1(<module>)
       2/1    0.000    0.000   18.956   18.956 /path/to/my/project/common/integrity/models/transactions.py:200(newfn)
       2/1    0.000    0.000   18.956   18.956 /path/to/my/project/common/integrity/models/transactions.py:134(recur)
       2/1    0.000    0.000   18.956   18.956 /usr/local/pypy/site-packages/django/db/transaction.py:210(inner)
       2/1    0.172    0.086   18.899   18.899 /path/to/my/project/common/integrity/tests/optimization.py:369(func_cached)
      9990    0.122    0.000   18.632    0.002 /usr/local/pypy/site-packages/django/db/models/manager.py:131(get)
      9990    0.127    0.000   16.638    0.002 /path/to/my/project/common/integrity/models/cache.py:1068(get)
      9990    0.073    0.000   12.478    0.001 /usr/local/pypy/site-packages/django/db/models/query.py:547(filter)
      9990    0.263    0.000   12.405    0.001 /path/to/my/project/common/integrity/models/cache.py:1047(_filter_or_exclude)
      9990    0.226    0.000   12.096    0.001 /usr/local/pypy/site-packages/django/db/models/query.py:561(_filter_or_exclude)
      9990    0.187    0.000    8.383    0.001 /path/to/my/project/common/integrity/models/cache.py:765(_clone)
      9990    0.212    0.000    7.662    0.001 /usr/local/pypy/site-packages/django/db/models/query.py:772(_clone)
      9990    1.025    0.000    7.125    0.001 /usr/local/pypy/site-packages/django/db/models/sql/query.py:226(clone)
129942/49972  1.674    0.000    6.021    0.000 /usr/local/pypy/lib-python/2.7/copy.py:145(deepcopy)
140575/110605 0.120    0.000    4.066    0.000 {len}
      9990    0.182    0.000    3.972    0.000 /usr/local/pypy/site-packages/django/db/models/query.py:74(__len__)
     19980    0.260    0.000    3.777    0.000 /path/to/my/project/common/integrity/models/cache.py:1062(iterator)
      9990    0.255    0.000    3.154    0.000 /usr/local/pypy/site-packages/django/db/models/sql/query.py:1149(add_q)
      9990    0.210    0.000    3.073    0.000 /path/to/my/project/common/integrity/models/cache.py:973(_query)
      9990    0.371    0.000    2.316    0.000 /usr/local/pypy/site-packages/django/db/models/sql/query.py:997(add_filter)
      9990    0.364    0.000    2.168    0.000 /path/to/my/project/common/integrity/models/cache.py:892(_deduct)
29974/9994    0.448    0.000    2.078    0.000 /usr/local/pypy/lib-python/2.7/copy.py:234(_deepcopy_tuple)
     19990    0.362    0.000    2.065    0.000 /path/to/my/project/common/integrity/models/cache.py:566(__init__)
     10000    0.086    0.000    1.874    0.000 /path/to/my/project/common/integrity/models/cache.py:1090(get_query_set)
     19990    0.269    0.000    1.703    0.000 /usr/local/pypy/site-packages/django/db/models/query.py:31(__init__)
      9990    0.122    0.000    1.643    0.000 /path/to/my/project/common/integrity/models/cache.py:836(_deduct_recur)
     19980    0.274    0.000    1.636    0.000 /usr/local/pypy/site-packages/django/utils/tree.py:55(__deepcopy__)
      9990    0.607    0.000    1.458    0.000 /path/to/my/project/common/integrity/models/cache.py:789(_deduct_local)
     10020    0.633    0.000    1.437    0.000 /usr/local/pypy/site-packages/django/db/models/sql/query.py:99(__init__)
    129942    0.841    0.000    1.191    0.000 /usr/local/pypy/lib-python/2.7/copy.py:267(_keep_alive)
 9994/9992    0.201    0.000    1.019    0.000 /usr/local/pypy/lib-python/2.7/copy.py:306(_reconstruct)

蟒蛇 2.7：

   3326403 function calls (3206359 primitive calls) in 12.430 CPU seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000   12.457   12.457 <string>:1(<module>)
        1    0.000    0.000   12.457   12.457 /path/to/my/project/common/integrity/models/transactions.py:200(newfn)
        1    0.000    0.000   12.457   12.457 /path/to/my/project/common/integrity/models/transactions.py:134(recur)
        1    0.000    0.000   12.457   12.457 /usr/local/lib/python2.7/dist-packages/django/db/transaction.py:210(inner)
        1    0.000    0.000   12.457   12.457 /path/to/my/project/common/integrity/models/transactions.py:165(recur2)
        1    0.089    0.089   12.450   12.450 /path/to/my/project/common/integrity/tests/optimization.py:369(func_cached)
     9990    0.198    0.000   12.269    0.001 /usr/local/lib/python2.7/dist-packages/django/db/models/manager.py:131(get)
     9990    0.087    0.000   11.281    0.001 /path/to/my/project/common/integrity/models/cache.py:1068(get)
     9990    0.040    0.000    8.161    0.001 /usr/local/lib/python2.7/dist-packages/django/db/models/query.py:547(filter)
     9990    0.110    0.000    8.121    0.001 /path/to/my/project/common/integrity/models/cache.py:1047(_filter_or_exclude)
     9990    0.127    0.000    7.983    0.001 /usr/local/lib/python2.7/dist-packages/django/db/models/query.py:561(_filter_or_exclude)
     9990    0.100    0.000    5.593    0.001 /path/to/my/project/common/integrity/models/cache.py:765(_clone)
     9990    0.122    0.000    5.125    0.001 /usr/local/lib/python2.7/dist-packages/django/db/models/query.py:772(_clone)
     9990    0.405    0.000    4.899    0.000 /usr/local/lib/python2.7/dist-packages/django/db/models/sql/query.py:226(clone)
129942/49972 1.456    0.000    4.505    0.000 /usr/lib/python2.7/copy.py:145(deepcopy)
129899/99929 0.191    0.000    3.117    0.000 {len}
     9990    0.111    0.000    2.968    0.000 /usr/local/lib/python2.7/dist-packages/django/db/models/query.py:74(__len__)
    19980    0.070    0.000    2.843    0.000 /path/to/my/project/common/integrity/models/cache.py:1062(iterator)
     9990    0.208    0.000    2.190    0.000 /path/to/my/project/common/integrity/models/cache.py:973(_query)
     9990    0.182    0.000    2.114    0.000 /usr/local/lib/python2.7/dist-packages/django/db/models/sql/query.py:1149(add_q)
19984/9994   0.291    0.000    1.644    0.000 /usr/lib/python2.7/copy.py:234(_deepcopy_tuple)
     9990    0.288    0.000    1.599    0.000 /usr/local/lib/python2.7/dist-packages/django/db/models/sql/query.py:997(add_filter)
     9990    0.171    0.000    1.454    0.000 /path/to/my/project/common/integrity/models/cache.py:892(_deduct)
    19980    0.177    0.000    1.208    0.000 /usr/local/lib/python2.7/dist-packages/django/utils/tree.py:55(__deepcopy__)
     9990    0.099    0.000    1.199    0.000 /path/to/my/project/common/integrity/models/cache.py:836(_deduct_recur)
     9990    0.349    0.000    1.040    0.000 /path/to/my/project/common/integrity/models/cache.py:789(_deduct_local)
4

1 回答 1

20

撇开 PyPy 对你的情况来说可能真的更慢的事实不谈，有一些因素可能会使其不必要地变慢：

  • 众所周知，Profiling 比 CPython 更慢了 PyPy。
  • 一些调试/记录代码可以禁用优化（例如，通过强制帧）。
  • 您正在使用的服务器可能是性能的主要因素（想想经典 CGI 与 JIT 的关系：它永远不会预热）。它也可以简单地影响结果（不同的 WSGI 服务器显示出不同的加速）。
  • 旧式类比新式类慢。
  • 即使所有内容都在内存中，您也可能会在 PyPy 的 SQLite 中遇到慢速路径。

您还可以查看JIT 友好性wiki 页面，了解更多关于什么会使 PyPy 变慢的提示。每晚构建可能也会更快，因为相对于 1.5 有许多改进。

对您的堆栈（服务器、操作系统、数据库）和设置（您如何进行基准测试？多少查询？）的更详细描述将使我们能够提供更好的答案。

于 2011-08-15T15:35:44.903 回答