0

从 Scrapy 结果中，标题中有一个不需要的非 ASCII 代码\u2013（又名character(150)或en dash），例如u'Director/Senior Director \u2013 Pathology'. 我正在尝试使用管道删除\u2013常规的,. 但是下面的代码不起作用。也不会报告错误消息。

from datetime import datetime
from hashlib import md5
from scrapy.exceptions import DropItem
from twisted.enterprise import adbapi
import re
import string

class ReplaceASC2InTitlePipeline(object):
"""replace unwanted ASCII characters in titles"""

ascii_to_filter = ["\u2013",]

def process_item(self, item, spider):
    for word in self.ascii_to_filter:
        desc = item.get('title')

        if (desc) and word in desc:
            spider.log("\u2013 in '%s' was replace" % (item['title']) )

            item['title']=item['title'].replace("\u2013", ",")
            return item
    else:
        return item
4

2 回答 2

0

在阅读了这篇 stackoverflow 帖子Replace non-ASCII characters...之后，我想出了这段代码，它将过滤掉标题中的所有非 ASCII 字符。对于我的情况，不需要非 ASCII 字符，所以它对我来说非常有效。

from datetime import datetime
from hashlib import md5
from scrapy.exceptions import DropItem
from twisted.enterprise import adbapi
import re
import string

class ReplaceASC2InTitlePipeline(object):
"""replace unwanted non-ASCII characters in titles"""

def process_item(self, item, spider):

    def remove_non_ascii(text):
        return ''.join(i for i in text if ord(i)<128)

    orig_titl = item.get('title')
    item['title'] = remove_non_ascii(orig_titl) 

    if item['title'] != orig_titl:
        spider.log("Non-ASCII character(s) was removed in '%s'" % (item['title']) )

    return item
于 2015-10-19T02:00:24.330 回答
0

"\u2013"应该是 unicode，所以只需替换：

ascii_to_filter = ["\u2013",]

和：

ascii_to_filter = [u"\u2013",]
于 2015-10-19T00:53:22.587 回答