1

我有许多大文件，有数千行 python dict 格式。我正在使用 json.dumps 将它们转换为 json 字符串。

import json
import ast

mydict = open('input', 'r')
output = open('output.json', "a")

for line in mydict:
        line = ast.literal_eval(line)
        line = json.dumps(line)
        output.write(line)
        output.write("\n")

这完美地工作，但是，它以单线程方式进行。有没有一种简单的方法可以利用系统中的剩余内核来加快速度？

编辑：

根据我在这里开始使用多处理库的建议：

import os
import json
import ast
from multiprocessing import Process, Pool

mydict = open('twosec.in', 'r')

def info(title):
        print title
        print 'module name:', __name__
        print 'parent process: ', os.getppid()
        print 'process id:', os.getpid()

def converter(name):
        info('converter function')
        output = open('twosec.out', "a")
        for line in mydict:
                line = ast.literal_eval(line)
                line = json.dumps(line)
                output.write(line)
                output.write("\n")

if __name__ == '__main__':
        info('main line')
        p = Process(target=converter, args=(mydict))
        p.start()
        p.join()

我不太明白Pool在哪里发挥作用，你能解释一下吗？

4

2 回答 2

2

我不知道有一种简单的方法可以让您从多线程中获得加速，但是如果您确实想要任何类型的加速，那么我建议您尝试使用ujson包而不是json. 它为我带来了非常显着的加速，基本上是免费的。json使用与使用常规包相同的方式。

http://pypi.python.org/pypi/ujson/

于 2012-04-20T18:45:44.880 回答
1

Wrap the code above in a function that takes as its single argument a filename and that writes the json to an output file.

Then create a Pool object from the multiprocessing module, and use Pool.map() to apply your function in parallel to the list of all files. This will automagically use all cores on your CPU, and because it uses multiple processes instead of threads, you won't run into the global interpreter lock.

Edit: Change the main portion of your program like so;

  if __name__ == '__main__':
     files = ['first.in', 'second.in', 'third.in'] # et cetera
     info('main line')
     p = Pool()
     p.map(convertor, files)
     p.close()

Of course you should also change convertor() to derive the output name from the input name!

Below is a complete example of a program to convert DICOM files into PNG format, using the ImageMagick program

"Convert DICOM files to PNG format, remove blank areas."

import os
import sys # voor argv.
import subprocess
from multiprocessing import Pool, Lock

def checkfor(args):
    try:
        subprocess.check_output(args, stderr=subprocess.STDOUT)
    except CalledProcessError:
        print "Required program '{}' not found! exiting.".format(progname)
        sys.exit(1)

def processfile(fname):
    size = '1574x2048'
    args = ['convert', fname, '-units', 'PixelsPerInch', '-density', '300', 
            '-crop', size+'+232+0', '-page', size+'+0+0', fname+'.png']
    rv = subprocess.call(args)
    globallock.acquire()
    if rv != 0:
        print "Error '{}' when processing file '{}'.".format(rv, fname)
    else:
        print "File '{}' processed.".format(fname)
    globallock.release()

## This is the main program ##
if __name__ == '__main__':
    if len(sys.argv) == 1:
        path, binary = os.path.split(sys.argv[0])
        print "Usage: {} [file ...]".format(binary)
        sys.exit(0)
    checkfor('convert')
    globallock = Lock()
    p = Pool()
    p.map(processfile, sys.argv[1:])
    p.close()
于 2012-04-20T19:10:47.400 回答