1

我正在尝试使用 Nokogiri 从多个供应商处获取商品的名称和价格。我通过方法参数将 CSS 选择器（查找名称和价格）传递给 Nokogiri。

有关如何将多个 URL 传递给“scrape”方法同时还传递其他参数（例如：供应商、i​​tem_path）的任何指导？还是我以完全错误的方式解决这个问题？

这是代码：

require 'rubygems' # Load Ruby Gems
require 'nokogiri' # Load Nokogiri
require 'open-uri' # Load Open-URI

@@collection = Array.new # Array to hold meta hash

def scrape(url, vendor, item_path, name_path, price_path)
    doc = Nokogiri::HTML(open(url)) # Opens URL
    items = doc.css(item_path) # Sets items
    items.each do |item| # Iterates through each item on grid
        @@collection << meta = Hash.new # Creates a new hash then add to global array
        meta[:vendor] = vendor
        meta[:name] = item.css(name_path).text.strip
        meta[:price] = item.css(price_path).to_s.scan(/\d+[.]\d+/).join 
    end
end

scrape( "page_a.html", "Sample Vendor A", "#products", ".title", ".prices")
scrape( ["page_a.html", "page_b.html"], "Sample Vendor B",  "#items", ".productname", ".price")
4

2 回答 2

1

url's您可以在第二个示例中以相同的方式传递多个：

scrape( ["page_a.html", "page_b.html"], "Sample Vendor B",  "#items", ".productname", ".price")

您的scrape方法将不得不遍历这些urls，例如：

def scrape(urls, vendor, item_path, name_path, price_path)
  urls.each do |url|
    doc = Nokogiri::HTML(open(url)) # Opens URL
    items = doc.css(item_path) # Sets items
    items.each do |item| # Iterates through each item on grid
        @@collection << meta = Hash.new # Creates a new hash then add to global array
        meta[:vendor] = vendor
        meta[:name] = item.css(name_path).text.strip
        meta[:price] = item.css(price_path).to_s.scan(/\d+[.]\d+/).join 
    end 
  end   
end

这也意味着第一个示例也需要作为数组传递：

scrape( ["page_a.html"], "Sample Vendor A", "#products", ".title", ".prices")
于 2013-03-16T18:48:29.583 回答
1

仅供参考，使用@@collection是不合适的。相反，编写您的方法以返回一个值：

def scrape(urls, vendor, item_path, name_path, price_path)
  collection = []
  urls.each do |url|
    doc = Nokogiri::HTML(open(url)) # Opens URL
    items = doc.css(item_path) # Sets items
    items.each do |item| # Iterates through each item on grid
      collection << {
        :vendor => vendor,
        :name   => item.css(name_path).text.strip,
        :price  => item.css(price_path).to_s.scan(/\d+[.]\d+/).join
      }
    end 
  end   

  collection
end

可以简化为：

def scrape(urls, vendor, item_path, name_path, price_path)
  urls.map { |url|
    doc = Nokogiri::HTML(open(url)) # Opens URL
    items = doc.css(item_path) # Sets items
    items.map { |item| # Iterates through each item on grid
      {
        :vendor => vendor,
        :name   => item.css(name_path).text.strip,
        :price  => item.css(price_path).to_s.scan(/\d+[.]\d+/).join
      }
    } 
  }
end
于 2013-03-17T06:44:20.663 回答