python - 使用python从非常大的文本文件（16gb）中跳过任何行的省时方法

Question

我有一个非常大的 16gb 文本文件。我需要跳过任何一行。我想以省时的方式跳过这些行。我正在使用 python 代码。如何做到这一点？

Answer 1

只需阅读您要跳过的行数并将其丢弃：

with open(your_file) as f_in:
    for i in range(number_of_lines_to_skip):
        f_in.readline()
    # your file is now at the line you want...  

您还可以使用enumerate生成器，一旦您跳过了您想要的行，它只会产生行：

with open(your_file) as f_in:
    for line in (line for i, line in enumerate(f_in) if i>lines_to_skip):
        # here only when you have skipped the first lines

第二个可能更快。

Answer 2

请注意，如果到达文件末尾，则调用next文件对象将引发。StopIteration

go_to_line_number = some_line_number

with open(very_large_file) as fp:

    for _ in range(go_to_line_number):
        next(fp)

    for line in fp:
        # start your work from desired line number
        pass