python - scrapy 输出标题和相关链接

Question

我的爬虫蜘蛛会显示所有网页的标题。请告诉我如何显示标题和与该标题相关的链接？我想解析这个页面。我的代码：

from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from scrapy.selector import HtmlXPathSelector
from probe1.items import SpiderItem

class SpiderSpider(CrawlSpider):
    name = "spider"
    allowed_domains = ["WEB_PAGE"]
    start_urls = [
    "http://www.WEB_PAGE"
    ]

    rules = (
        Rule(
            SgmlLinkExtractor(allow_domains=("WEB_PAGE",)),
            callback='parse_page', follow=True
        ),
    )


    def parse_page(self, response):
      hxs = HtmlXPathSelector(response)
      print hxs
      sites = hxs.select('//title')
      items = []
      for s in sites:
      item = SpiderItem()
          item['title'] = s.select('//title').extract
          items.append(item)
      return items   

Answer 1

response.url包含您需要的内容：

网址

包含响应 URL 的字符串。