0

我有一个循环来找到几个总和：

for t in reversed(range(len(inputs))):
  dy = np.copy(ps[t])
  dy[targets[t]] -= 1 
  dWhy += np.dot(dy, hs[t].T)
  dby += dy

输入值太大，我必须让它平行。所以我把循环转换为一个单独的函数。我尝试使用 ThreadPoolExecutor，但与顺序算法相比，结果时间很慢。

这是我最小的工作示例：

import numpy as np
import concurrent.futures
import time, random 

from concurrent.futures import ThreadPoolExecutor
import threading

#parameters
dWhy = np.random.sample(300)
dby = np.random.sample(300)

def Func(ps, targets, hs,  t):
  global dWhy, dby
  dy = np.copy(ps[t])
  dWhy += np.dot(dy, hs[t].T)
  dby += dy

  return dWhy, dby

if __name__ == '__main__':    

    ps = np.random.sample(100000)
    targets = np.random.sample(100000)
    hs = np.random.sample(100000)

    start = time.time()

    for t in range(100000):
        dy = np.copy(ps[t])
        dWhy += np.dot(dy, hs[t].T)
        dby += dy

    finish = time.time()
    print("One thread: ")
    print(finish-start)

    dWhy = np.random.sample(300)
    dby = np.random.sample(300)
    start = time.time()

    with concurrent.futures.ThreadPoolExecutor() as executor:
        args = ((ps, targets, hs,  t) for t in range(100000))
        for out1, out2  in executor.map(lambda p: Func(*p), args):
            dWhy, dby = out1, out2

    finish = time.time()
    print("Multithreads time: ")
    print(finish-start)

在我的 PC 上一个线程时间 ~ 3 秒，多线程时间 ~ 1 分钟。

4

2 回答 2

0

将 lambda 转换为命名函数。

于 2018-11-24T12:50:20.137 回答
0

考虑用广播来暗示它：

import numpy as np
dWhy = np.random.sample(300)
dby = np.random.sample(300)

ps = np.random.sample(100000)
targets = np.random.sample(100000)
hs = np.random.sample(100000)

dWhy += np.dot(ps,hs)
dby += np.sum(ps)

运行速度快 20000 倍

timer = time.time()
for i in range(20000):
    dWhy += np.dot(ps,hs)
    dby += np.sum(ps)
print(time.time()-timer)
>>3.2034592628479004
print(time.time()-timer)/20000)
>>0.00016017296314239503
于 2018-11-26T12:39:31.657 回答