2

我想加载以下十六进制文件

1) 第一行的初始化值 (IV)，
2) 第二行的加密密钥，
3) 第三行的明文数量，以及
4) 密码块链接 (CBC) 模式下 AES 加密的实际明文

进入numpy数组。

6bce1cb8d64153f82570751b6653c943
b15a65475a91774a45106fbc28f0df70
10
f493befb2dcad5118d523a4a4bf4a504
54fc4e0a82ae8dc56cc7befc9994b79d
878d287647b457fd95d40691b6e0c8ab
dc0adc16665eb96a15d3257752ae67dc
8cda3b8f23d38e9240b9a89587f69970
e06301763146c1bac24619e61015f481
c19def2f12e5707d89539e18ad104937
048d734a1a36d4346edc7ceda07ff171
5e621ce0a570478c1c2ec3e557ca3e0d
e55c57b119ff922b7f87db0ead2006cd

如果文件的一致性让您感到困扰，您可以忽略第三行，它告诉您要加密的纯文本的数量。除第三行外的所有行都是 128 位十六进制条目

这个想法是将此文件加载到 numpy 数组中，然后有效地进行 AES 加密。

我如何将其加载到 numpy 数组中，然后使用 Crypto.Cipher 中的 AES 对此文件和类似文件进行 AES 加密。我有这种格式的文件，其中包含多达 1 亿个纯文本。

谢谢，如果您有任何问题，请告诉我

4

1 回答 1

4

我假设您想要 unhexlify 数据，并将生成的字节串存储为固定长度的字符串，而不是object. （您不能将它们存储为某种 int128 类型，因为 numpy 没有这种类型。）

为了避免将 3.2GB 的文本读入内存，并使用大致相同的数量将其预处理为所需的形式，您可能想要使用fromiter，所以：

with open(myfile) as f:
    iv = binascii.unhexlify(f.readline().strip())
    key = binascii.unhexlify(f.readline().strip())
    count = int(f.readline())
    a = np.fromiter((binascii.unhexlify(line.strip()) for line in f), dtype='|S16')

如果您有 10GB 的 RAM 可用（粗略的猜测），将整个内容读取为of可能会更快，然后将其转换两次……但我对此表示怀疑。arrayobject

As to whether this will help… You might get a little benefit, because AES-ing 16 bytes may be fast enough that the cost of iteration is noticeable. Let's test it and see.

With 64-bit Mac Python 2.7.2, I created an array of 100000 S16s by copying your example repeatedly. Then:

In [514]: %timeit [aes.encrypt(x) for x in a]
10 loops, best of 3: 166 ms per loop
In [515]: %timeit np.vectorize(aes.encrypt)(a)
10 loops, best of 3: 126 ms per loop

So, that's almost a 25% savings. Not bad.

Of course the array takes longer to build than just keeping things in an iterator in the first place—but even taking that into account, there's still a 9% performance gain. And it may well be reasonable to trade 1.6GB for a 9% speedup in your use case.

Keep in mind that I'm only building an array of 100K objects out of a pre-existing 100K list; with 100M objects read off disk, I/O is probably going to become a serious factor, and it's quite possible that iterative processing (which allows you to interleave the CPU costs with disk waits) will do a whole lot better.

In other words, you need to test with your own real data and scenario. But you already knew that.

For a wider variety of implementations, with a simple perf-testing scaffolding, see this pastebin.

You might want to try combining different approaches. For example, you can use the grouper recipe from itertools to batch things into, say, 32K plaintexts at a time, then process each batch with numpy, to get the best of both. And then pool.imap that numpy processing, to get the best of all 3. Or, alternatively, put the one big numpy array into shared memory, and make each multiprocessing task process a slice of that array.

于 2013-05-30T18:20:49.190 回答