2

I have a python code which does some manipulation of data contained in a Shapefile.

Among others stuffs, the code does this:

 xxx=0
 for i in channels:
      ptsi=mat(shapes[i].points)
      xx = ptsi[:,0]
      yy = ptsi[:,1]

      nanx=argwhere(isnan(xx))      
      nany=argwhere(isnan(yy))

      if (nanx == nany and len(nanx) != 0):
          xx[nanx] = []
          yy[nany] = []
      elif (len(nanx) != 0 or len(nany) != 0):
          xx = []
          yy = []

      if (len(xx) > 0):          
       yyy = 0 
       dd = str(i*100/N_channels) + '%\r\r'
      # os.write(1,dd)
       dist = zeros(len(xx)-1)

       dist = dist_min
       for j in channels:
           pts=mat(shapes[j].points)
           xx2 = pts[:,0]
           yy2 = pts[:,1]

           nanx=argwhere(isnan(xx2))          
           nany=argwhere(isnan(yy2))

           if (nanx == nany and len(nanx) != 0):
            xx2[nanx] = []
            yy2[nany] = []
           elif (len(nanx) != 0 or len(nany) != 0):
            xx2 = []
            yy2 = []
           if (len(xx2) > 0):  
            if (i != j): 
               ds = gaa(xx,yy,xx2[0],yy2[0])
               de = gaa(xx,yy,xx2[-1],yy2[-1])
               nande = ~isnan(de)
               nands = ~isnan(ds)
               fe = np.argwhere(de<=dist)
               fs = np.argwhere(ds<=dist)
               nozeroe = array(where(de != 0))
               nozeroe = nozeroe[0]
               nozeros = array(where(ds != 0))
               nozeros = nozeros[0] 
               if(fs.size >0):
                    iis=array(np.argwhere(array(ds==min(ds))))

                    iis = iis.flatten()
                    iis = iis[0]
                    p1 = xxx + iis 
                    p2 = yyy
                    G.add_edge(p2,p1,weight=w_con)

               elif (fe.size > 0):
                    iie=array(np.argwhere(array(de==min(de))))
                    iie = iie.flatten()
                    iie = iie[0]
                    p1 = xxx + iie
                    p2 = yyy  + len(pts) -1
                    G.add_edge(p2,p1,weight=w_con)

           yyy = yyy + len(pts)

      xxx = xxx + len(ptsi)

Basically, the code scans over polylines and search for a minimal distance in order to merge then in a common graph (using Networkx). This part works fine except that this is really slow as I deal with more than 100's of thousand of objects (It takes roughly 20 hours in its current version).

These embedded loops are not efficient, so I'd like to know, if using multithreading could be helpful and if so, how can I modify this very part of the code? I'm fine with CUDA or OpenCL if it can help.

Thanks for any feedback!

4

2 回答 2

1

由于Global Interpreter Lock，Python 代码无法通过多线程充分利用多核，需要使用多处理才能充分利用多核。

于 2012-11-13T03:03:14.527 回答
0

More threads will not help; more processes might, but if you can use CUDA, that would probably be a good move. A thread is just a way to allow multiple things to share a processes CPU time, it does not speed up slow code, but will just slow you down more.

于 2012-11-13T02:53:46.613 回答