我使用 twisted 及其 deferredGenerator 编写了一个爬虫。以下是关于我的问题的代码:
@defer.deferredGenerator
def getReviewsFromPage(self,title,params):
def deferred1(page):
d = defer.Deferred()
reactor.callLater(1,d.callback,self.parseReviewJson(page))
return d
def deferred2(dataL,title):
d = defer.Deferred()
reactor.callLater(1,d.callback,self.writeToCSV(dataL,title=title))
return d
cp = 1
#for cp in range(1,15000):
while self.running:
print cp
params["currentPageNum"] = cp
url = self.generateReviewUrl(self.urlPrefix,params = params)
print url
wfd = defer.waitForDeferred(getPage(url,timeout=10))
yield wfd
page = wfd.getResult()
wfd = defer.waitForDeferred(deferred1(page))
yield wfd
dataList = wfd.getResult()
wfd = defer.waitForDeferred(deferred2(dataList,title))
yield wfd
cp = cp+1
我使用发电机
self.getReviewsFromPage(title,params)
reactor.run()
我的问题是:当函数'getPage'出现超时错误时,我该怎么做才能处理错误并再次抓取错误页面?我曾经在getPage中添加了一个addErrback并想调用getPage,但似乎当reactor运行时,它不会再接收新事件了。
你们中有人遇到过同样的问题吗?我很感激你的帮助