我有一个名为 bontyurls.csv 的文件,如下所示:
http://bontrager.com/model/11383
http://bontrager.com/model/01740
http://bontrager.com/model/09595
我希望我的脚本读取该文件,然后吐出这样的文件:bonty_test_urls_results.csv
url,model_names
http://bontrager.com/model/11383,"Road TLR Conversion Kit"
http://bontrager.com/model/01740,"404 File Not Found"
http://bontrager.com/model/09595,"RXL Road"
这是我到目前为止所得到的:
# based on code from here: http://www.andrewsturges.com/2011/09/how-to-harvest-web-data-using-ruby-and.html
require 'nokogiri'
require 'open-uri'
require 'csv'
@urls = Array.new
@model_names = Array.new
urls = CSV.read("bontyurls.csv")
(0..urls.length - 1).each do |index|
puts urls[index][0]
doc = Nokogiri::HTML(open(urls[index][0]))
doc.xpath('//h1').each do |model_name|
@model_name << model_name.content
end
end
# write results to file
CSV.open("bonty_test_urls_results.csv", "wb") do |row|
row << ["url", "model_names"]
(0..@urls.length - 1).each do |index|
row << [
@urls[index],
@model_names[index]]
end
end
该代码不起作用。我收到此错误:
$ ruby bonty_test_urls.rb
http://bontrager.com/model/00310
bonty_test_urls.rb:15:in `block (2 levels) in <main>': undefined method `<<' for nil:NilClass (NoMethodError)
from /home/simon/.rvm/gems/ruby-1.9.3-p194/gems/nokogiri-1.5.5/lib/nokogiri/xml/node_set.rb:239:in `block in each'
from /home/simon/.rvm/gems/ruby-1.9.3-p194/gems/nokogiri-1.5.5/lib/nokogiri/xml/node_set.rb:238:in `upto'
from /home/simon/.rvm/gems/ruby-1.9.3-p194/gems/nokogiri-1.5.5/lib/nokogiri/xml/node_set.rb:238:in `each'
from bonty_test_urls.rb:14:in `block in <main>'
from bonty_test_urls.rb:11:in `each'
from bonty_test_urls.rb:11:in `<main>'
这是一些至少返回 model_name 的代码。我只是无法让它在更大的脚本中工作:
require 'open-uri'
require 'nokogiri'
doc = Nokogiri::HTML(open("http://bontrager.com/model/09124"))
doc.xpath('//h1').each do |node|
puts node.text
end
另外,我还没有弄清楚如何处理返回 404 的 URL。