3

我正在尝试查看 Dask 是否适合我的项目，并编写了一些非常简单的测试用例来研究它的性能。但是，Dask 需要相对较长的时间来简单地执行延迟初始化。

@delayed
def normd(st):
    return st.lower().replace(',', '')

@delayed
def add_vald(v):
    return v+5

def norm(st):
    return st.lower().replace(',', '')

def add_val(v):
    return v+5

test_list = [i for i in range(1000)]
test_list1 = ["AeBe,oF,221e"]*1000

%timeit rlist = [add_val(y) for y in test_list]
#124 µs ± 7.25 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

%timeit rlist = [norm(y) for y in test_list1]
#392 µs ± 18.9 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

%timeit rlist = [add_vald(y) for y in test_list]
#19.1 ms ± 436 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

rlist = [add_vald(y) for y in test_list]
%timeit rlist1 = compute(*rlist, get=dask.multiprocessing.get)
#892 ms ± 36.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

%timeit rlist = [normd(y) for y in test_list1]
#18.7 ms ± 408 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

rlist = [normd(y) for y in test_list1]
%timeit rlist1 = compute(*rlist, get=dask.multiprocessing.get)
#912 ms ± 54.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

我已经研究了Dask For Loop In Parallel和parallel dask for loop 比常规循环慢？我尝试将大小增加到 100 万个项目，但是虽然常规循环大约需要一秒钟，但 dask 循环永远不会结束。在等待半小时完成 add_vald 的延迟初始化后，我将其杀死。

我不确定这里出了什么问题，非常感谢您提供的任何见解。谢谢！

4

0 回答 0