0

我有一个脚本，它使用 glob 读取目录中的多个文件，然后根据在特定 json 字段中的每一行上找到的日期将它们逐行拆分为新文件：

这是一个有效的脚本：

import json
import glob
import fileinput
from dateutil import parser
import ast
import gzip

line = []

filestobeanalyzed = glob.glob('../data/*')

for fileName in filestobeanalyzed:
        inputfilename = fileName
        print inputfilename
        for line in fileinput.input([inputfilename]):
                line = line.strip();
                if not line: continue
                line = ast.literal_eval(line)
                line = json.dumps(line)
                if not json.loads(line).get('created_at'): continue
                date = json.loads(line).get('created_at')
                date_converted = parser.parse(date).strftime('%Y%m%d')
                outputfilename = gzip.open(date_converted, "a")
                outputfilename.write(line)
                outputfilename.write("\n")

outputfilename.close()

到达目录中第一个文件的末尾时出现以下错误：

python split_json_docs_by_date_with_dict-to-json.py 
../data/research_data_p1.json
Traceback (most recent call last):
  File "split_json_docs_by_date_with_dict-to-json.py", line 18, in <module>                                                                                  
    line = ast.literal_eval(line)                                                                                                                            
  File "/usr/lib64/python2.7/ast.py", line 49, in literal_eval                                                                                               
    node_or_string = parse(node_or_string, mode='eval')                                                                                                      
  File "/usr/lib64/python2.7/ast.py", line 37, in parse                                                                                                      
    return compile(source, filename, mode, PyCF_ONLY_AST)                                                                                                    
  File "<unknown>", line 1                                                                                                                                   
    {u'user': {u'follow_request_sent': None, u'profile_use_background_image': False, u'default_profile_image': False, u'geo_enabled': False, u'verified': False, u'profile_image_url_https': u'https://si0.twimg.com/profile_images/1829421396/yA6hEz2j_normal', u'profile_sidebar_fill_color': u'DDEEF6', u'id': 15054232, u'profile_text_color': u'333333', u'followers_count': 117, u'protected': False, u'id_str': u'15054232', u'profile_background_color': u'858585', u'listed_count': 6, u'utc_offset': -25200, u'statuses_count': 9418, u'description': u"Hi- I'm Jordan, and I refuse to put any effort into this bio. Well... except just enough to type this I guess.", u'friends_count': 59, u'location': u'Washington Terrace, UT', u'profile_link_color': u'0084B4', u'profile_image_url': u'http://a3.twimg.com/profile_images/1829421396/yA6hEz2j_normal', u'notifications': N

对我来说很明显 ast 无法评估该行，因为它不完整但是如果我插入：

if not ast.literal_eval(line): continue

之前：

line = ast.literal_eval(line)

我仍然得到完全相同的错误。

4

1 回答 1

2

如果您只是想忽略错误，那么您可以在 try 块中捕获异常并继续。如果您想解析多行 JSON，那么 fileinput 可能不是最佳选择。

这是一个适合您的编辑。它包含基本的多行 JSON 支持和 try 块，因此无法解析的代码不会使程序崩溃。这是未经测试的，因为我无权访问您的测试数据。只需删除带有注释的行即可删除基本的多行 JSON 支持。

import json
import glob
import fileinput
from dateutil import parser
import ast
import gzip

line = []

filestobeanalyzed = glob.glob('../data/*')

for fileName in filestobeanalyzed:
        inputfilename = fileName
        print inputfilename
        pastlines = "" # stores previous unparsable lines
        for line in fileinput.input([inputfilename]):
                line = pastlines + line # put past unparsable lines with current line
                line = line.strip();
                if not line: continue
                try:
                    line = ast.literal_eval(line)
                    line = json.dumps(line)
                    pastlines = "" # reset unparsable lines
                except:
                    pastlines += line # add current line to unparsable lines
                    continue
                date = json.loads(line).get('created_at', None)
                if not date: continue
                date_converted = parser.parse(date).strftime('%Y%m%d')
                outputfilename = gzip.open(date_converted, "a")
                outputfilename.write(line)
                outputfilename.write("\n")

outputfilename.close()
于 2012-06-05T01:04:48.643 回答