0

我将一个 1MM 行和 35 列的 pandas 数据框对象推送到 ipython 并行引擎的 directView 中。但是，我无法将此数据（甚至是空数据帧）推送到引擎中，因为我的函数无法打印数据帧的长度。这是我的代码片段。

ipcluster start -n 4

def myfn():
  rc = Client()
  dview = rc[:]
  data = ..... #queried from some source of 1MM rows
  dview.push(dict(data=data,new=DataFrame()))
  async = dview.map_async(f,range(3))

  return async 

def f(n):
  test = DataFrame() 
  x = len(data) # type data = pandas.core.frame.DataFrame
  #print len(test) #works fine, gets three "0"s
  #print len(new)  # empty DF, gets an error below
  print len(data)  # 1MM row DF, gets an error below
  return x

查看 asyn.stdout 后，这是我收到的错误。任何帮助表示赞赏！：

In [205]: x1.stdout
Out[205]:
[u'Traceback (most recent call last):\n  File "/myProj/ipython/0.13.2-py27/lib/IPython/core/ultratb.py", line 760, in structured_traceback\n    records = _fixed_getinnerframes(etb, context, tb_offset)\n  File "/myProj/ipython/0.13.2-py27/lib/IPython/core/ultratb.py", line 242, in _fixed_getinnerframes\n    records  = fix_frame_records_filenames(inspect.getinnerframes(etb, context))\n  File "//myProj/core/2.7.3-64/lib/python2.7/inspect.py", line 1043, in getinnerframes\n    framelist.append((tb.tb_frame,) + getframeinfo(tb, context))\n  File "//myProj/core/2.7.3-64/lib/python2.7/inspect.py", line 1007, in getframeinfo\n    lines, lnum = findsource(frame)\n  File "//myProj/core/2.7.3-64/lib/python2.7/inspect.py", line 580, in findsource\n    if pat.match(lines[lnum]): break\nIndexError: list index out of range\n',
 u'Traceback (most recent call last):\n  File "/myProj/ipython/0.13.2-py27/lib/IPython/core/ultratb.py", line 760, in structured_traceback\n    records = _fixed_getinnerframes(etb, context, tb_offset)\n  File "/myProj/ipython/0.13.2-py27/lib/IPython/core/ultratb.py", line 242, in _fixed_getinnerframes\n    records  = fix_frame_records_filenames(inspect.getinnerframes(etb, context))\n  File "//myProj/core/2.7.3-64/lib/python2.7/inspect.py", line 1043, in getinnerframes\n    framelist.append((tb.tb_frame,) + getframeinfo(tb, context))\n  File "//myProj/core/2.7.3-64/lib/python2.7/inspect.py", line 1007, in getframeinfo\n    lines, lnum = findsource(frame)\n  File "//myProj/core/2.7.3-64/lib/python2.7/inspect.py", line 580, in findsource\n    if pat.match(lines[lnum]): break\nIndexError: list index out of range\n',
 u'Traceback (most recent call last):\n  File "/myProj/ipython/0.13.2-py27/lib/IPython/core/ultratb.py", line 760, in structured_traceback\n    records = _fixed_getinnerframes(etb, context, tb_offset)\n  File "/myProj/ipython/0.13.2-py27/lib/IPython/core/ultratb.py", line 242, in _fixed_getinnerframes\n    records  = fix_frame_records_filenames(inspect.getinnerframes(etb, context))\n  File "//myProj/core/2.7.3-64/lib/python2.7/inspect.py", line 1043, in getinnerframes\n    framelist.append((tb.tb_frame,) + getframeinfo(tb, context))\n  File "//myProj/core/2.7.3-64/lib/python2.7/inspect.py", line 1007, in getframeinfo\n    lines, lnum = findsource(frame)\n  File "//myProj/core/2.7.3-64/lib/python2.7/inspect.py", line 580, in findsource\n    if pat.match(lines[lnum]): break\nIndexError: list index out of range\n']
4

1 回答 1

1

IPython 0.13 中有一个错误导致 DataFrame 序列化失败。它在 IPython 1.0 中已修复，因此问题应该通过升级来解决。如果由于某种原因无法升级，那么您将不得不自己序列化 DataFrame，最简单的方法是在将对象交给 IPython 之前进行酸洗，然后在另一边取消酸洗。显然，如果可能的话，只升级会更好。

于 2013-11-19T05:45:08.137 回答