1

I want to pickle my objects compressed with snappy-python. Because these objects are large I want to use something streaming -- i.e. pickle.dump, not pickle.dumps.

Unfortunately, the snappy and pickle APIs don't seem particularly compatible:

snappy.stream_compress(src, dst, blocksize=65536)
pickle.dump(obj, file, protocol=None)

Any thoughts on what magic I would need for something like snappy.stream_compress(pickle.dump_magic(obj), dst) to work similarly to what pickle.dump(obj, dst) does today?

4

1 回答 1

1

最简单的方法是先到pickle.dump文件，然后再到snappy.stream_compress文件。

def snappy_pickle_dump(obj, f):
    tmpf = ...
    with open(tmpf, 'wb') as F:
        pickle.dump(obj, F)
    with open(f, 'wb') as OUT:
        with open(tmpf, 'rb') as IN:
            snappy.stream_compress(IN, OUT)
    os.remove(tmpf)

如果您必须避免存储整个未压缩的转储，您可以让一个线程pickle.dump将对象放入一个StringIO对象中，而另一个线程snappy.stream_compress将其放入。

于 2013-03-28T20:07:39.177 回答