1

我正在开发一个 MCTS 算法，我试图通过并行推出多个叶子来并行化一些工作。在进行了一批推出之后，我想返回并将我的结果添加到树中（撤消我的虚拟损失），然后再选择另一批叶子进行推出。除了速度之外，这一切正常 - 我发现围绕 ProcessPoolExecutor 上下文的连续循环变慢了。代码部分如下：

for _ in range(8):
   tick = time.time()
   with concurrent.futures.ProcessPoolExecutor() as executor:
       value_estimates = executor.map(NeuralNet.evaluate, leaves, chunksize=round(batch_size/8)+1)
   tock = time.time()

然后我得到以下计时结果。

Took 2.3520002365112305 sec to run 500 times

Took 2.5691237449645996 sec to run 500 times

Took 2.8875749111175537 sec to run 500 times

Took 3.2885916233062744 sec to run 500 times

Took 3.43363618850708 sec to run 500 times

Took 3.6769683361053467 sec to run 500 times

Took 3.948704719543457 sec to run 500 times

Took 4.299146890640259 sec to run 500 times

同样的模式发生在较大的循环中，所花费的时间只是继续增长和增长。

这是什么原因？有没有办法让每个循环以相同的速度执行？

提前致谢

4

1 回答 1

0

我会建议使用库多处理。也许会很有趣。

import numpy as np from concurrent.futures._base import as_completed import concurrent.futures import time from multiprocessing import Pool, cpu_count

data = np.random.randint(0, 1000000, 100000000) N = 10000



def label_data(i):
    print(f'Started executing iteration {i} for data of length {len(data)}')
    #print(f'Started executing iteration {i} for data of length ')

t0= time.clock() futures = {} with concurrent.futures.ProcessPoolExecutor() as executor:
    for i in range(N):
        executor.submit(label_data, i)

for future in as_completed(futures):
    iteration = futures[future] t1 = time.clock() - t0 print("Time elapsed: ", t1) # CPU seconds elapsed (floating point) time_concurrent
= t1 """ multiprocessing.cpu_count()-1  """


t0= time.clock() with Pool(cpu_count-1) as p:
    p.map(label_data, [i for i in range(0,N)]) t1 = time.clock() - t0 print("Time elapsed: ", t1) # CPU seconds elapsed (floating point) time_multiprocessing = t1

print("------------------------------") print("Concurrent library time: ", time_concurrent) print("Multiprocessing library time: ", time_multiprocessing)

这是该示例中的一些结果

Concurrent library time:  4.678165
Multiprocessing librar

y时间：0.017128999999999728

于 2021-01-30T22:20:36.703 回答