我有两个文件:
- metadata.csv:包含一个 ID,后跟供应商名称、文件名等
- hashes.csv:包含一个 ID,后跟一个哈希 ID 本质上是一种外键,将文件元数据与其哈希相关联。
我编写了这个脚本来快速提取与特定供应商相关的所有哈希值。它在完成处理 hashes.csv 之前就崩溃了
stored_ids = []
# this file is about 1 MB
entries = csv.reader(open(options.entries, "rb"))
for row in entries:
# row[2] is the vendor
if row[2] == options.vendor:
# row[0] is the ID
stored_ids.append(row[0])
# this file is 1 GB
hashes = open(options.hashes, "rb")
# I iteratively read the file here,
# just in case the csv module doesn't do this.
for line in hashes:
# not sure if stored_ids contains strings or ints here...
# this probably isn't the problem though
if line.split(",")[0] in stored_ids:
# if its one of the IDs we're looking for, print the file and hash to STDOUT
print "%s,%s" % (line.split(",")[2], line.split(",")[4])
hashes.close()
该脚本在停止之前通过 hashes.csv 获取大约 2000 个条目。我究竟做错了什么?我以为我正在逐行处理它。
附言。csv 文件是流行的 HashKeeper 格式,我正在解析的文件是 NSRL 哈希集。http://www.nsrl.nist.gov/Downloads.htm#converter
更新:下面的工作解决方案。感谢所有评论的人!
entries = csv.reader(open(options.entries, "rb"))
stored_ids = dict((row[0],1) for row in entries if row[2] == options.vendor)
hashes = csv.reader(open(options.hashes, "rb"))
matches = dict((row[2], row[4]) for row in hashes if row[0] in stored_ids)
for k, v in matches.iteritems():
print "%s,%s" % (k, v)