0

我有一系列需要处理的 JSON 消息,并且在字符编码方面遇到了一些问题。我从 RabbitMQ 队列中检索此消息,如下所示:

@data = []
@queue.pop do |metadata, payload|
  parsed_message = JSON.parse(payload)
  @data << parsed_message
end

然后我使用 Resque 处理它们并将它们排入队列,如下所示:

Resque.enqueue(Worker, @data)

当我运行工人时,它会引发此错误:

** [16:06:13 2012-09-14] 5375: Error reserving job: #<Encoding::InvalidByteSequenceError: "\xC3" on US-ASCII>
** [16:06:13 2012-09-14] 5375: /usr/local/lib/ruby/1.9.1/json/common.rb:148:in `encode'
/usr/local/lib/ruby/1.9.1/json/common.rb:148:in `initialize'
/usr/local/lib/ruby/1.9.1/json/common.rb:148:in `new'
/usr/local/lib/ruby/1.9.1/json/common.rb:148:in `parse'
/usr/local/lib/ruby/gems/1.9.1/gems/multi_json-1.3.6/lib/multi_json/adapters/json_common.rb:7:in `load'
/usr/local/lib/ruby/gems/1.9.1/gems/multi_json-1.3.6/lib/multi_json.rb:93:in `load'
/usr/local/lib/ruby/gems/1.9.1/gems/resque-1.22.0/lib/resque/helpers.rb:38:in `decode'
/usr/local/lib/ruby/gems/1.9.1/gems/resque-1.22.0/lib/resque.rb:149:in `pop'
/usr/local/lib/ruby/gems/1.9.1/gems/resque-1.22.0/lib/resque/job.rb:100:in `reserve'
/usr/local/lib/ruby/gems/1.9.1/gems/resque-1.22.0/lib/resque.rb:303:in `reserve'
/usr/local/lib/ruby/gems/1.9.1/gems/resque-1.22.0/lib/resque/worker.rb:209:in `block in reserve'
/usr/local/lib/ruby/gems/1.9.1/gems/resque-1.22.0/lib/resque/worker.rb:207:in `each'
/usr/local/lib/ruby/gems/1.9.1/gems/resque-1.22.0/lib/resque/worker.rb:207:in `reserve'
/usr/local/lib/ruby/gems/1.9.1/gems/resque-1.22.0/lib/resque/worker.rb:136:in `block in work'
/usr/local/lib/ruby/gems/1.9.1/gems/resque-1.22.0/lib/resque/worker.rb:133:in `loop'
/usr/local/lib/ruby/gems/1.9.1/gems/resque-1.22.0/lib/resque/worker.rb:133:in `work'
/usr/local/lib/ruby/gems/1.9.1/gems/resque-1.22.0/lib/resque/tasks.rb:36:in `block (2 levels) in <top (required)>'
/usr/local/lib/ruby/1.9.1/rake/task.rb:205:in `call'
/usr/local/lib/ruby/1.9.1/rake/task.rb:205:in `block in execute'
/usr/local/lib/ruby/1.9.1/rake/task.rb:200:in `each'
/usr/local/lib/ruby/1.9.1/rake/task.rb:200:in `execute'
/usr/local/lib/ruby/1.9.1/rake/task.rb:158:in `block in invoke_with_call_chain'
/usr/local/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
/usr/local/lib/ruby/1.9.1/rake/task.rb:151:in `invoke_with_call_chain'
/usr/local/lib/ruby/1.9.1/rake/task.rb:144:in `invoke'
/usr/local/lib/ruby/1.9.1/rake/application.rb:116:in `invoke_task'
/usr/local/lib/ruby/1.9.1/rake/application.rb:94:in `block (2 levels) in top_level'
/usr/local/lib/ruby/1.9.1/rake/application.rb:94:in `each'
/usr/local/lib/ruby/1.9.1/rake/application.rb:94:in `block in top_level'
/usr/local/lib/ruby/1.9.1/rake/application.rb:133:in `standard_exception_handling'
/usr/local/lib/ruby/1.9.1/rake/application.rb:88:in `top_level'
/usr/local/lib/ruby/1.9.1/rake/application.rb:66:in `block in run'
/usr/local/lib/ruby/1.9.1/rake/application.rb:133:in `standard_exception_handling'
/usr/local/lib/ruby/1.9.1/rake/application.rb:63:in `run'
/usr/local/bin/rake:32:in `<main>'
rake aborted!

我已将工作人员的执行函数中的所有代码包装在开始/救援块中,以尝试捕获异常但无济于事,这使我认为它来自 Resque 内部的某个地方。

我尝试使用原始数据将 Resque 工作人员排队,然后在工作人员内部将其解析为 JSON 并设置 UTF-8 编码:

raw_data.each do |msg|
  msg = msg.encode("UTF-8", :invalid => :replace, :undef => :replace, :replace => "")                        
  parsed_msg = JSON.parse(msg) rescue nil                                                                     
  next if parsed_msg.nil?                                                                                     
  parsed_data << parsed_msg                                                                                   
 end

我仍然得到同样的错误。它并不总是同一个字符,如果我没记错的话,有时它是 \xC2 或 \xD8

有任何想法吗?

4

1 回答 1

1

文件编码是否可能不正确?

尝试插入:

# encoding: UTF-8

在相关 ruby​​ 源文件的顶部。这是 Ruby 1.9 中最令人讨厌的事情之一。

于 2012-09-18T20:48:20.843 回答