0

我在 Windows Vista 64 位上使用 Python.org 版本 2.7 64 位。我有以下 Scrapy 代码，其中我定义 SgmlLinkExtractor 的方式没有正确抓取网站：

from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from scrapy.selector import Selector
from scrapy.item import Item
from scrapy.spider import BaseSpider
from scrapy import log
from scrapy.cmdline import execute
from scrapy.utils.markup import remove_tags
import time


class ExampleSpider(CrawlSpider):
    name = "goal3"
    allowed_domains = ["whoscored.com"]
    start_urls = ["http://www.whoscored.com"]
    download_delay = 1
    #rules = [Rule(SgmlLinkExtractor(allow=()), 
                  #follow=True),
             #Rule(SgmlLinkExtractor(allow=()), callback='parse_item')
    #]
    rules = [
        Rule(
            SgmlLinkExtractor(allow=('Regions/252/Tournaments/2',)), 
            callback='parse_item',
            follow=True,
        )
    ]
    def parse_item(self,response):
        self.log('A response from %s just arrived!' % response.url)
        scripts = response.selector.xpath("normalize-space(//title)")
        for scripts in scripts:
            body = response.xpath('//p').extract()
            body2 = "".join(body)
            print remove_tags(body2).encode('utf-8')  


execute(['scrapy','crawl','goal3'])

我已经尝试了 SgmlLinkExtractor 定义的几个不同版本，但似乎都打印到 Command Shell 如下：

    Contact Us | About Us | Glossary | Privacy Policy | WhoScored Ratings
            Copyright ┬® 2014 WhoScored.com
2014-07-20 00:14:38+0100 [goal3] DEBUG: Filtered duplicate request: <GET http://www.whoscored.com/Statistics> - no more duplicates will be shown (see DUPEFILTER_DEBUG to show all duplicates)
2014-07-20 00:14:40+0100 [goal3] DEBUG: Crawled (200) <GET http://www.whoscored.com/Statistics/Teams> (referer: http://www.whoscored.com/Statistics)
2014-07-20 00:14:40+0100 [goal3] DEBUG: A response from http://www.whoscored.com/Statistics/Teams just arrived!

                Contact Us | About Us | Glossary | Privacy Policy | WhoScored Ratings
            Copyright ┬® 2014 WhoScored.com
2014-07-20 00:14:41+0100 [goal3] DEBUG: Redirecting (302) to <GET http://www.whoscored.com/404.html?aspxerrorpath=/Statistics/3> from <GET http://www.whoscored.com/Statistics/3>
2014-07-20 00:14:42+0100 [goal3] DEBUG: Redirecting (302) to <GET http://www.whoscored.com/404.html?aspxerrorpath=/Statistics/2> from <GET http://www.whoscored.com/Statistics/2>
2014-07-20 00:14:43+0100 [goal3] DEBUG: Redirecting (302) to <GET http://www.whoscored.com/404.html?aspxerrorpath=/Statistics/1> from <GET http://www.whoscored.com/Statistics/1>
2014-07-20 00:14:45+0100 [goal3] DEBUG: Crawled (200) <GET http://www.whoscored.com/404.html?aspxerrorpath=/Statistics/3> (referer: http://www.whoscored.com/Statistics/Teams)
2014-07-20 00:14:45+0100 [goal3] DEBUG: A response from http://www.whoscored.com/404.html?aspxerrorpath=/Statistics/3 just arrived!

2014-07-20 00:14:46+0100 [goal3] DEBUG: Crawled (200) <GET http://www.whoscored.com/404.html?aspxerrorpath=/Statistics/2> (referer: http://www.whoscored.com/Statistics/Teams)
2014-07-20 00:14:46+0100 [goal3] DEBUG: A response from http://www.whoscored.com/404.html?aspxerrorpath=/Statistics/2 just arrived!

2014-07-20 00:14:47+0100 [goal3] DEBUG: Crawled (200) <GET http://www.whoscored.com/404.html?aspxerrorpath=/Statistics/1> (referer: http://www.whoscored.com/Statistics/Teams)
2014-07-20 00:14:47+0100 [goal3] DEBUG: A response from http://www.whoscored.com/404.html?aspxerrorpath=/Statistics/1 just arrived!

任何人都可以在这里看到任何明显的东西为什么这不起作用？

谢谢

4

0 回答 0