python - 为什么使用 pickle.HIGHEST_PROTOCOL 转储和加载需要更长的时间？

Answer 1

当文档谈到“新型类”时，它（可能）指的是用户定义的新型类。如果你用它们做一个简单的基准测试，你会发现协议 2 在转储它们时比协议 0 快两倍：

>>> import cPickle
>>> import timeit
>>> class MyObject(object):
...     def __init__(self, val):
...             self.val = val
...     def method(self):
...             print self.val
... 
>>> timeit.timeit('cPickle.dumps(MyObject(100), 0)', 'from __main__ import cPickle, MyObject')
17.654622077941895
>>> timeit.timeit('cPickle.dumps(MyObject(100), 1)', 'from __main__ import cPickle, MyObject')
14.536609172821045
>>> timeit.timeit('cPickle.dumps(MyObject(100), 2)', 'from __main__ import cPickle, MyObject')
8.885567903518677

加载也会导致 2 倍的加速：

>>> dumped = cPickle.dumps(MyObject(100), 0)
>>> timeit.timeit('cPickle.loads(dumped)', 'from __main__ import cPickle, dumped')
4.6161839962005615
>>> dumped = cPickle.dumps(MyObject(100), 1)
>>> timeit.timeit('cPickle.loads(dumped)', 'from __main__ import cPickle, dumped')
4.351701021194458
>>> dumped = cPickle.dumps(MyObject(100), 2)
>>> timeit.timeit('cPickle.loads(dumped)', 'from __main__ import cPickle, dumped')
2.3936450481414795

在您的特殊情况下，情况可能相反，但如果没有定义fetch_sku2spu_dict等的代码，我们就不能说什么。我唯一可以假设的是返回值是 a dict，但在这种情况下，协议 2 的速度大约快 6 倍：

>>> mydict = dict(zip(range(100), range(100)))
>>> timeit.timeit('cPickle.dumps(mydict, 0)', 'from __main__ import cPickle, mydict')
46.335021018981934
>>> timeit.timeit('cPickle.dumps(mydict, 1)', 'from __main__ import cPickle, mydict')
7.913743019104004
>>> timeit.timeit('cPickle.dumps(mydict, 2)', 'from __main__ import cPickle, mydict')
7.798863172531128

加载速度大约快 2.5 倍：

>>> dumped = cPickle.dumps(mydict, 0)
>>> timeit.timeit('cPickle.loads(dumped)', 'from __main__ import cPickle, dumped')
32.81050395965576
>>> dumped = cPickle.dumps(mydict, 1)
>>> timeit.timeit('cPickle.loads(dumped)', 'from __main__ import cPickle, dumped')
13.997781038284302
>>> dumped = cPickle.dumps(mydict, 2)
>>> timeit.timeit('cPickle.loads(dumped)', 'from __main__ import cPickle, dumped')
14.006750106811523

---

另一方面，当使用模块的python版本时，我发现：

>>> mydict = dict(zip(range(100), range(100)))
>>> timeit.timeit('pickle.dumps(mydict,0)', 'from __main__ import pickle, mydict', number=10000)
2.9552500247955322
>>> timeit.timeit('pickle.dumps(mydict,1)', 'from __main__ import pickle, mydict', number=10000)
3.831756830215454
>>> timeit.timeit('pickle.dumps(mydict,2)', 'from __main__ import pickle, mydict', number=10000)
3.842888116836548

因此，使用协议 1 和 2 转储内置对象似乎比在 python 版本中使用协议 0 慢。但是在加载对象时，协议 0 再次是三个中最慢的：

>>> dumped = pickle.dumps(mydict, 0)
>>> timeit.timeit('pickle.loads(dumped)', 'from __main__ import pickle, dumped', number=10000)
2.988792896270752
>>> dumped = pickle.dumps(mydict, 1)
>>> timeit.timeit('pickle.loads(dumped)', 'from __main__ import pickle, dumped', number=10000)
1.2793281078338623
>>> dumped = pickle.dumps(mydict, 2)
>>> timeit.timeit('pickle.loads(dumped)', 'from __main__ import pickle, dumped', number=10000)
1.5425071716308594

正如您在上面的迷你基准中看到的那样，腌制所需的时间取决于许多因素，从您正在腌制的对象类型到您使用的腌制模块的版本。如果没有进一步的信息，我们将无法解释为什么在您的情况下协议 2 慢得多。