我已经困惑了很长时间,似乎无法弄清楚如何解决它。最容易提供工作虚拟代码:
require(RCurl)
require(XML)
#set a bunch of options for curl
options(RCurlOptions = list(cainfo = system.file("CurlSSL", "cacert.pem", package = "RCurl")))
agent="Firefox/23.0"
curl = getCurlHandle()
curlSetOpt(
cookiejar = 'cookies.txt' ,
useragent = agent,
followlocation = TRUE ,
autoreferer = TRUE ,
httpauth = 1L, # "basic" http authorization version -- this seems to make a difference for India servers
curl = curl
)
list1 <- c('http://timesofindia.indiatimes.com//articleshow/2933112.cms',
'http://timesofindia.indiatimes.com//articleshow/2933131.cms',
'http://timesofindia.indiatimes.com//articleshow/2933209.cms',
'http://timesofindia.indiatimes.com//articleshow/2933277.cms')
#note list2 has a new link inserted in 2nd position; this is the link that kills the following getURL calls
list2 <- c('http://timesofindia.indiatimes.com//articleshow/2933112.cms',
'http://timesofindia.indiatimes.com//articleshow/2933019.cms',
'http://timesofindia.indiatimes.com//articleshow/2933131.cms',
'http://timesofindia.indiatimes.com//articleshow/2933209.cms',
'http://timesofindia.indiatimes.com//articleshow/2933277.cms')
for ( i in seq( list1 ) ){
print(list1[i])
html <-
try( getURL(
list1[i],
maxredirs = as.integer(20),
followlocation = TRUE,
curl = curl
),TRUE)
if (class (html) == "try-error") {
print(paste("error accessing",list1[i]))
rm(html)
gc()
next
} else {
print('success')
}
}
gc()
for ( i in seq( list2 ) ){
print(list2[i])
html <-
try( getURL(
list2[i],
maxredirs = as.integer(20),
followlocation = TRUE,
curl = curl
),TRUE)
if (class (html) == "try-error") {
print(paste("error accessing",list2[i]))
rm(html)
gc()
next
} else {
print('success')
}
}
这应该能够在安装了 RCurl 和 XML 库的情况下运行。关键是当我插入http://timesofindia.indiatimes.com//articleshow/2933019.cms
列表中的第二个位置时,它会杀死循环其余部分的成功(其他链接相同)。当链接包含 PDF(检查以查看)时,会发生这种情况(在这种情况和其他情况下一致)。
有关如何解决此问题的任何想法,以便获取包含 PDF 的链接不会杀死我的循环?如您所见,我试图清除可能有问题的对象,gc()
到处都是,等等,但我无法弄清楚为什么 PDF 会杀死我的循环。
谢谢!
只是为了检查,这是我的两个for
循环的输出:
#[1] "http://timesofindia.indiatimes.com//articleshow/2933112.cms"
#[1] "success"
#[1] "http://timesofindia.indiatimes.com//articleshow/2933131.cms"
#[1] "success"
#[1] "http://timesofindia.indiatimes.com//articleshow/2933209.cms"
#[1] "success"
#[1] "http://timesofindia.indiatimes.com//articleshow/2933277.cms"
#[1] "success"
和
#[1] "http://timesofindia.indiatimes.com//articleshow/2933112.cms"
#[1] "success"
#[1] "http://timesofindia.indiatimes.com//articleshow/2933019.cms"
#[1] "error accessing http://timesofindia.indiatimes.com//articleshow/2933019.cms"
#[1] "http://timesofindia.indiatimes.com//articleshow/2933131.cms"
#[1] "error accessing http://timesofindia.indiatimes.com//articleshow/2933131.cms"
#[1] "http://timesofindia.indiatimes.com//articleshow/2933209.cms"
#[1] "error accessing http://timesofindia.indiatimes.com//articleshow/2933209.cms"
#[1] "http://timesofindia.indiatimes.com//articleshow/2933277.cms"
#[1] "error accessing http://timesofindia.indiatimes.com//articleshow/2933277.cms"