0

我是新手ipyparallel，我想使用这个包来实现我的机器学习应用程序的并行计算。

下面是对 的测试ipyparallel，我add在 func.py 文件中定义了一个函数，在 test.py 文件中定义了 main 函数。

func.py 的代码是：

#!/usr/bin/env python
# coding=utf-8

def add(*numbers):
    numbers = list(numbers)
    for i, n in enumerate(numbers):
        numbers[i] = n + 1
    return numbers

test.py 的代码是：

#!/usr/bin/env python
# coding=utf-8

from func import add
from ipyparallel import Client

if __name__ == '__main__':
    rc = Client(
        '/home/fit/.ipython/profile_default/security/ipcontroller-client.json')

    print map(add, [1, 2, 3]
    print rc[0].map_sync(add, [1, 2, 3, 4])

既然你知道map可以正常运行，但是当运行时map_sync，命令行返回：

☁  test  python test.py 
[[2], [3], [4]]
Traceback (most recent call last):
  File "test.py", line 14, in <module>
    print rc[0].map_sync(add, [1, 2, 3, 4])
  File "/usr/local/lib/python2.7/dist-packages/ipyparallel/client/view.py", line 353, in map_sync
    return self.map(f,*sequences,**kwargs)
  File "<string>", line 2, in map
  File "/usr/local/lib/python2.7/dist-packages/ipyparallel/client/view.py", line 54, in sync_results
    ret = f(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/ipyparallel/client/view.py", line 618, in map
    return pf.map(*sequences)
  File "/usr/local/lib/python2.7/dist-packages/ipyparallel/client/remotefunction.py", line 268, in map
    ret = self(*sequences)
  File "<string>", line 2, in __call__
  File "/usr/local/lib/python2.7/dist-packages/ipyparallel/client/remotefunction.py", line 75, in sync_view_results
    return f(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/ipyparallel/client/remotefunction.py", line 251, in __call__
    return r.get()
  File "/usr/local/lib/python2.7/dist-packages/ipyparallel/client/asyncresult.py", line 104, in get
    raise self._exception
ipyparallel.error.CompositeError: one or more exceptions from call to method: add
[0:apply]: ImportError: No module named func

如果我在 test.py 文件中定义函数，map_sync可以运行：

#!/usr/bin/env python
# coding=utf-8

#from func import add
from ipyparallel import Client

def add(*numbers):
    numbers = list(numbers)
    for i, n in enumerate(numbers):
        numbers[i] = n + 1
    return numbers


if __name__ == '__main__':
    rc = Client(
        '/home/fit/.ipython/profile_default/security/ipcontroller-client.json')

    print map(add, [1, 2, 3])

    print rc[0].map_sync(add, [1, 2, 3, 4])

结果是：

☁  test  python test.py
[[2], [3], [4]]
[[2], [3], [4], [5]]

我想知道map_sync如何使用其他文件中定义的函数？以及如何导入这些功能？由于from py_file import func不适用于map_sync.

4

1 回答 1

0

需要的模块应该被复制（或者，您可以推送或模块）到engine machines并安装 3 方包engine machines，如果没有，ImportError将会发生。

但是，在运行程序时，您应该运行：

$ ipcontroller --ip=client_ip --reuse

on client machine，目录下会生成2个/home/user/.ipython/profile_default/security文件

$ ls /home/user/.ipython/profile_default/security 
ipcontroller-client.json  ipcontroller-engine.json

因此，两者ipcontroller-client.json和都ipcontroller-engine.json必须复制到engine machines并运行

$ ipengine --file=/path/to/ipcontroller-engine.json

on engine machines，这样parallel computing环境就搭建好了。

接下来，您可以定义您的parallel computing任务并运行您的程序。

于 2015-09-29T03:50:18.483 回答