1

我有一个 Ruby 脚本在 1.8.7 中运行良好，但现在在 1.9.3 中因 Ruby 错误而崩溃。

它使用一个简单的线程池来打开一个 URL 列表，以便它们被预先缓存。

以下是脚本的相关部分。

线程池：

require 'thread'

# see http://burgestrand.se/articles/quick-and-simple-ruby-thread-pool.html
class Pool
  def initialize(size)
    @size = size
    @jobs = Queue.new

    @pool = Array.new(@size) do |i|
      Thread.new do
        Thread.current[:id] = i

        catch(:exit) do
          loop do
            job, args = @jobs.pop
            job.call(*args)
          end
        end
      end
    end
  end

  def schedule(*args, &block)
    @jobs << [block, args]
  end

  def shutdown
    @size.times do
      schedule { throw :exit }
    end

    @pool.map(&:join)
  end
end

load_url 方法：

def load_url(url)
  http_proxy = ENV['http_proxy']
  proxy_regex = /http\:\/\/(\w+):(\w+)@([\w|\.]+):(\d+)/

  url_regex = /^((http[s]?|ftp):\/)?\/?([^:\/\s]+)((\/\w+)*\/)([\w\-\.]+[^#?\s]+)(.*)?(#[\w\-]+)?$/

  url.match(url_regex)

  host = $3
  path = $4 + $6

  if http_proxy and http_proxy =~ proxy_regex
    proxy_addr = $3
    proxy_port = $4
    proxy_user = $1 
    proxy_pass = $2

    Net::HTTP::Proxy(proxy_addr, proxy_port, proxy_user, proxy_pass).start(host) do |http|
      req = Net::HTTP::Get.new(path)
      http.read_timeout = 500
      response = http.request(req)
    end
  else
    Net::HTTP.start(host) do |http|
      req = Net::HTTP::Get.new(path)
      http.read_timeout = 500
      response = http.request(req)
    end
  end
end

下载代码：

pool = Pool.new(5)

# the_trips are loaded from the db
the_trips.each do |trip|
  cleaned_urls = TripPage.empty_cache(trip)

  cleaned_urls.each do |cleaned_url|
    pool.schedule do
      puts "Loading #{cleaned_url}..."
      begin
        load_url(cleaned_url)
        puts "#{cleaned_url} OK\n"
      rescue Timeout::Error => e
        puts "Caught Timeout error for #{cleaned_url} :("
      end
    end
  end
end

# wait for all threads to finish
puts "Done, shutting down the threadpool..."
pool.shutdown

设置就这么多。

当我用 Ruby 1.9.3 运行它时，我得到了这个：

Loading http://some.server.com/reisen/2010003...
Loading http://some.server.com/reisen/2010003-the-title...
Loading http://some.server.com/reisen/2010011...
Loading http://some.server.com/reisen/2010011-different-title...
Loading http://some.server.com/reisen/2010100...
http://some.server.com/reisen/2010003 OK
Loading http://some.server.com/reisen/2010100-yet-another-title...
~/.rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/net/protocol.rb:160: [BUG] probable buffer overflow: 17 for 0
ruby 1.9.3p0 (2011-10-30 revision 33570) [i386-solaris2.10]

-- Control frame information -----------------------------------------------
c:0025 p:0017 s:0112 b:0112 l:000111 d:000111 METHOD ~/.rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/net/protocol.rb:160
c:0024 p:0061 s:0107 b:0107 l:000106 d:000106 METHOD ~/.rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/net/protocol.rb:124
c:0023 p:0159 s:0101 b:0101 l:000100 d:000100 METHOD ~/.rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/net/http.rb:2584
c:0022 p:0024 s:0091 b:0091 l:000090 d:000090 METHOD ~/.rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/net/http.rb:2669
c:0021 p:0106 s:0086 b:0086 l:000077 d:000085 BLOCK  ~/.rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/net/http.rb:1321
c:0020 p:---- s:0084 b:0084 l:000083 d:000083 FINISH
c:0019 p:---- s:0082 b:0082 l:000081 d:000081 CFUNC  :catch
c:0018 p:0025 s:0078 b:0078 l:000077 d:000077 METHOD ~/.rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/net/http.rb:1316
c:0017 p:0097 s:0072 b:0072 l:000071 d:000071 METHOD ~/.rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/net/http.rb:1293
c:0016 p:0053 s:0065 b:0065 l:001d94 d:000064 BLOCK  clear_diskcache.rb:60
c:0015 p:0057 s:0060 b:0060 l:000059 d:000059 METHOD ~/.rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/net/http.rb:745
c:0014 p:0182 s:0057 b:0057 l:000056 d:000056 METHOD ~/.rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/net/http.rb:557
c:0013 p:0140 s:0044 b:0044 l:001d94 d:001d94 METHOD clear_diskcache.rb:57
c:0012 p:0034 s:0031 b:0031 l:0012e4 d:000030 BLOCK  clear_diskcache.rb:115
c:0011 p:---- s:0028 b:0028 l:000027 d:000027 FINISH
c:0010 p:---- s:0026 b:0026 l:000025 d:000025 CFUNC  :call
c:0009 p:0034 s:0023 b:0023 l:002110 d:000022 BLOCK  clear_diskcache.rb:16
c:0008 p:---- s:0019 b:0019 l:000018 d:000018 FINISH
c:0007 p:---- s:0017 b:0017 l:000016 d:000016 CFUNC  :loop
c:0006 p:0009 s:0014 b:0014 l:002110 d:000013 BLOCK  clear_diskcache.rb:14
c:0005 p:---- s:0012 b:0012 l:000011 d:000011 FINISH
c:0004 p:---- s:0010 b:0010 l:000009 d:000009 CFUNC  :catch
c:0003 p:0038 s:0006 b:0006 l:002110 d:000005 BLOCK  clear_diskcache.rb:13
c:0002 p:---- s:0004 b:0004 l:000003 d:000003 FINISH
c:0001 p:---- s:0002 b:0002 l:000001 d:000001 TOP   

-- Ruby level backtrace information ----------------------------------------
clear_diskcache.rb:13:in `block (2 levels) in initialize'
clear_diskcache.rb:13http://some.server.com/reisen/2010011 OK
:in `catch'
clear_diskcache.rb:14:in `block (3 levels) in initialize'
clear_diskcache.rbLoading http://some.server.com/reisen/201T004...
:14:in `loop'
clear_diskcache.rb:16:in `block (4 levels) in initialize'
clear_diskcache.rb:16:in `call'
clear_diskcache.rb:113:in `block (4 levels) in <top (required)>'
clear_diskcache.rb:113:in `puts'
clear_diskcache.rb:113:in `puts'
clear_diskcache.rb:113:in `write'

知道可能是什么问题吗？我找到了这个，但我不能 100% 确定这是否是我看到的错误：http: //bugs.ruby-lang.org/issues/6099

如果是这样，看起来好像直到 Ruby 2.0 才会发布修复程序......

4

0 回答 0