为什么 nokogiri 在服务器很忙并且我正在逐页请求页面时等待几秒(3-5),但是当这些请求处于循环状态时,nokogiri 不会等待并抛出超时消息。我正在使用超时块包装请求,但 nokogiri 根本不等待那个时间。有什么建议的程序吗?
# this is a method from the eng class
def get_page(url,page_type)
begin
timeout(10) do
# Get a Nokogiri::HTML::Document for the page we’re interested in...
@@doc = Nokogiri::HTML(open(url))
end
rescue Timeout::Error
puts "Time out connection request"
raise
end
end
# this is a snippet from the main app calling eng class
# receives a hash with urls and goes throgh asking one by one
def retrieve_in_loop(links)
(0..links.length).each do |idx|
url = links[idx]
puts "Visiting link #{idx} of #{links.length}"
puts "link: #{url}"
begin
@@eng.get_page(url, product)
rescue Exception => e
puts "Error getting url: #{idx} #{url}"
puts "This link will be skeeped. Continuing with next one"
end
end
end