1

我有一棵非常大的树（>~1M），我需要随机访问它的一些元素并使用一些标准检索数据。问题是当我想在我的计算机内核之间并行化这个任务时。经过一些调试，我意识到问题是通过文件对象（f=ROOT.TFile("MergedTrees.root")) 在每个进程中运行。这是我应该在每个内核中运行的代码：

def worker(tree,f,func,table,features,indexes):
    print "starting:" , mp.current_process().name
    time.sleep(2)

    print tree.GetEntry(100) # *****returns -1,means I/O error************* 

    x=[]
    y=[]
    appendx=x.append
    appendy=y.append
    for ind in indexes[0]:
       # print type(ind)
       # print tree.GetEntry(100)
        tree.GetEntry(ind) #access event number "ind"
        appendx(func(tree,features)) #get data samples
        appendy(table[ind]) #store lable of current event

    return x,y

这就是我调用函数的方式：

results=[pool.apply_async(worker,args=(self.Tree,self.f,self.func,self.table,features,(ch,))) for ch in parts]
        pool.close()
        pool.join()

你对解决这个问题有什么建议吗？一个简单的方法是拥有我的根文件的多个副本，每个进程都打开特定的文件。但我不喜欢这个解决方案，而且看起来很傻。

例如，如果我print f在我的函数中尝试，我会收到此错误：

 *** Break *** segmentation violation



===========================================================
There was a crash.
This is the entire stack trace of all threads:
===========================================================
#0  0x00007f25bce4912d in waitpid () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f25bcddbe8e in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007f25bcddc2a0 in system () from /lib/x86_64-linux-gnu/libc.so.6
#3  0x00007f25b008298e in TUnixSystem::StackTrace() () from /home/jadidi/Downloads/root/lib/libCore.so
#4  0x00007f25b0082223 in TUnixSystem::DispatchSignals(ESignals) () from /home/jadidi/Downloads/root/lib/libCore.so
#5  <signal handler called>
#6  0x00007f25b2b6d9e5 in TDirectoryFile::Get(char const*) () from /home/jadidi/Downloads/root/lib/libRIO.so
#7  0x00007f25b00cc485 in G__G__Base1_8_0_33(G__value*, char const*, G__param*, int) () from /home/jadidi/Downloads/root/lib/libCore.so
#8  0x00007f25af52112f in Cint::G__CallFunc::Execute(void*) () from /home/jadidi/Downloads/root/lib/libCint.so
#9  0x00007f25b2f446cb in PyROOT::TRootObjectExecutor::Execute(Cint::G__CallFunc*, void*) () from /home/jadidi/Downloads/root/lib/libPyROOT.so
#10 0x00007f25b2f4b08e in PyROOT::TMethodHolder<PyROOT::TScopeAdapter, PyROOT::TMemberAdapter>::CallSafe(void*) () from /home/jadidi/Downloads/root/lib/libPyROOT.so
#11 0x00007f25b2f4c1eb in PyROOT::TMethodHolder<PyROOT::TScopeAdapter, PyROOT::TMemberAdapter>::Execute(void*) () from /home/jadidi/Downloads/root/lib/libPyROOT.so
#12 0x00007f25b2f48b72 in PyROOT::TMethodHolder<PyROOT::TScopeAdapter, PyROOT::TMemberAdapter>::operator()(PyROOT::ObjectProxy*, _object*, _object*, long) () from /home/jadidi/Downloads/root/lib/libPyROOT.so
#13 0x00007f25b2f4e616 in PyROOT::(anonymous namespace)::mp_call(PyROOT::MethodProxy*, _object*, _object*) () from /home/jadidi/Downloads/root/lib/libPyROOT.so
#14 0x0000000000460462 in PyObject_CallFunctionObjArgs ()
#15 0x00000000004dd34b in ?? ()
#16 0x000000000048e36a in ?? ()
#17 0x000000000047d159 in PyObject_HasAttr ()
#18 0x00007f25b2f4ff54 in PyROOT::(anonymous namespace)::op_repr(PyROOT::ObjectProxy*) () from /home/jadidi/Downloads/root/lib/libPyROOT.so
#19 0x000000000047cb0c in _PyObject_Str ()
#20 0x000000000047cbea in PyObject_Str ()
#21 0x00000000004d644c in ?? ()
#22 0x00000000004cfe81 in PyFile_WriteObject ()
#23 0x000000000049bb2e in PyEval_EvalFrameEx ()
#24 0x000000000049fd55 in PyEval_EvalCodeEx ()
#25 0x00000000004c7608 in ?? ()
#26 0x000000000045fed4 in PyObject_Call ()
#27 0x000000000049af25 in PyEval_EvalFrameEx ()
#28 0x000000000049fd55 in PyEval_EvalCodeEx ()
#29 0x00000000004c7608 in ?? ()
#30 0x000000000045fed4 in PyObject_Call ()
#31 0x000000000049af25 in PyEval_EvalFrameEx ()
#32 0x000000000049982f in PyEval_EvalFrameEx ()
#33 0x000000000049982f in PyEval_EvalFrameEx ()
#34 0x000000000049fd55 in PyEval_EvalCodeEx ()
#35 0x00000000004c7436 in ?? ()
#36 0x000000000045fed4 in PyObject_Call ()
#37 0x0000000000461b0f in ?? ()
#38 0x000000000045fed4 in PyObject_Call ()
#39 0x000000000048dc2f in ?? ()
#40 0x000000000048bf6d in ?? ()
#41 0x000000000045fed4 in PyObject_Call ()
#42 0x00000000004996be in PyEval_EvalFrameEx ()
#43 0x000000000049982f in PyEval_EvalFrameEx ()
#44 0x000000000049982f in PyEval_EvalFrameEx ()
#45 0x000000000049fd55 in PyEval_EvalCodeEx ()
#46 0x00000000004c7436 in ?? ()
#47 0x000000000045fed4 in PyObject_Call ()
#48 0x0000000000461b0f in ?? ()
#49 0x000000000045fed4 in PyObject_Call ()
#50 0x000000000048dc2f in ?? ()
#51 0x000000000048bf6d in ?? ()
#52 0x000000000045fed4 in PyObject_Call ()
#53 0x00000000004996be in PyEval_EvalFrameEx ()
#54 0x000000000049fd55 in PyEval_EvalCodeEx ()
#55 0x0000000000499502 in PyEval_EvalFrameEx ()
#56 0x000000000049fd55 in PyEval_EvalCodeEx ()
#57 0x0000000000499502 in PyEval_EvalFrameEx ()
#58 0x000000000049fd55 in PyEval_EvalCodeEx ()
#59 0x00000000004eeb92 in PyEval_EvalCode ()
#60 0x000000000049c4d5 in PyEval_EvalFrameEx ()
#61 0x000000000049fd55 in PyEval_EvalCodeEx ()
#62 0x0000000000499502 in PyEval_EvalFrameEx ()
#63 0x000000000049fd55 in PyEval_EvalCodeEx ()
#64 0x0000000000499502 in PyEval_EvalFrameEx ()
#65 0x000000000049fd55 in PyEval_EvalCodeEx ()
#66 0x0000000000499502 in PyEval_EvalFrameEx ()
#67 0x000000000049fd55 in PyEval_EvalCodeEx ()
#68 0x0000000000499502 in PyEval_EvalFrameEx ()
#69 0x000000000049fd55 in PyEval_EvalCodeEx ()
#70 0x0000000000499502 in PyEval_EvalFrameEx ()
#71 0x000000000049fd55 in PyEval_EvalCodeEx ()
#72 0x0000000000499502 in PyEval_EvalFrameEx ()
#73 0x000000000049fd55 in PyEval_EvalCodeEx ()
#74 0x0000000000499502 in PyEval_EvalFrameEx ()
#75 0x000000000049fd55 in PyEval_EvalCodeEx ()
#76 0x00000000004eeb92 in PyEval_EvalCode ()
#77 0x00000000004ff7f4 in ?? ()
#78 0x000000000042cdd0 in PyRun_FileExFlags ()
#79 0x000000000042d798 in PyRun_SimpleFileExFlags ()
#80 0x0000000000418db5 in Py_Main ()
#81 0x00007f25bcdb7eff in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6
#82 0x00000000004c91b1 in _start ()

我希望有人能把我引向正确的方向

PS：我刚开始使用多处理，所以在解释相关内容时请更具体一点。谢谢

4

0 回答 0