0

我正在做一个虚拟内存模拟器，但我遇到了问题。我需要从 k(4) 个文件中读取 n (8) 行，例如：我读取文件 1 - 文件 2 - 文件 3 - 文件 4 的前 8 行，然后我再次从每个文件中读取第 9 - 17 行, 直到每个文件用完行。

我对文件的输入没有问题，并且已经完成了这段代码。

def rr_process(quantum, file, fline):
    global rr_list #List to save the reading lines
    condition = file_len(file) #Return the total lines of passed file
    with open(file) as fp:
        line = fp.readlines() #Save all the lines of the file in a list   
        for i in range(fline,fline+quantum): #for i in range(NewStartLine, NewStartLie + n_lines)
            if i <= condition-1:
                sline = line[i].rstrip()#Remove /n from lines
                rr_list.append(sline) #append the n_lines to the list
            else:
                break 

operation = concat_count//(n_proc*quantum) #total_lines//(k_files*n_lines)

for i in range(0,operation):
    for fname in process: #Open each file (4)
         rr_process(quantum,fname,fline) #Calls the read lines function
    fline = fline + quantum + 1 #New start line number 0-9-17...

我根本没有成功，我需要读取 50k 行但我的程序只读取 44446。代码中的错误是什么？或者有什么更好的方法来处理这个？多谢你们！

4

2 回答 2

1

使用模块文档提供的grouper和函数，这可以减少到几行代码。roundrobinitertools

import contextlib
from itertools import zip_longest, cycle, islice, chain

# Define grouper() and roundrobin() here

with contextlib.ExitStack() as stack:
    # Open each file *once*; the exit stack will make sure they get closed
    files = [stack.enter_context(open(fname)) for frame in process]
    # Instead of iterating over each file line by line, we'll iterate
    # over them in 8-line batches.
    groups = [grouper(f, 8) for f in files]
    # Interleave the groups by taking an 8-line group from one file,
    # then another, etc.
    interleaved = roundrobin(*groups)
    # *Then* flatten them into a stream of single lines
    flattened = chain.from_iterable(interleaved)
    # Filter out the None padding added by grouper() and
    # read the lines into a list
    lines = list(filter(lambda x: x is not None, flattened))

请注意，在您调用 之前list，您实际上并没有从文件中读取任何内容；您只是在构建一个功能管道，将按需处理输入。

作为参考，这些是文档grouper的定义和roundrobin复制而来。

# From itertools documentation
def grouper(iterable, n, fillvalue=None):
    "Collect data into fixed-length chunks or blocks"
    # grouper('ABCDEFG', 3, 'x') --> ABC DEF Gxx"
    args = [iter(iterable)] * n
    return zip_longest(*args, fillvalue=fillvalue)


# From itertools documentation
def roundrobin(*iterables):
    "roundrobin('ABC', 'D', 'EF') --> A D E B F C"
    # Recipe credited to George Sakkis
    num_active = len(iterables)
    nexts = cycle(iter(it).__next__ for it in iterables)
    while num_active:
        try:
            for next in nexts:
                yield next()
        except StopIteration:
            # Remove the iterator we just exhausted from the cycle.
            num_active -= 1
            nexts = cycle(islice(nexts, num_active))
于 2019-04-18T16:15:10.037 回答
0

我最终得到了与chepner非常相似的东西......

首先，我们定义了一个简单的文件，它遍历文件中的行，将它们分组为块：

def read_blocks(path, nlines):
    with open(path) as fd:
        out = []
        for line in fd:
            out.append(line)
            if len(out) == nlines:
                yield out
                out = []
        if out:
            yield out

然后我定义了一个函数，它将一组迭代器的输出交错（即与roundrobinchepner 相同，我发现版本itertools有些不透明）：

def interleave(*iterables):
    iterables = [iter(it) for it in iterables]
    i = 0
    while iterables:
        try:
            yield next(iterables[i])
        except StopIteration:
            del iterables[i]
        else:
            i += 1
        if i >= len(iterables):
            i = 0

然后我们定义一个函数将上述内容放在一起：

def read_files_in_blocks(filenames, nlines):
    return interleave(*(read_blocks(path, nlines) for path in filenames))

并用一些虚拟数据调用它：

filenames = ['foo.txt', 'bar.txt', 'baz.txt']

for block in read_files_in_blocks(filenames, 5):
    for line in block:
        print(line)
于 2019-04-18T16:34:19.840 回答