3

我有一个带有两种不同编码的大文件。“主”文件是 UTF-8，但某些字符（如<80>（isoxxx 中的€）或<9F>（isoxxx 中的ß））采用 ISO-8859-1 编码。我可以用它来替换无效字符：

 string.encode("iso8859-1", "utf-8", {:invalid => :replace, :replace => "-"}).encode("utf-8")

问题是，我需要这个错误的编码字符，所以替换为“-”对我来说没用。如何使用 ruby​​ 修复文档中错误的编码字符？

编辑：我试过这个:fallback选项，但没有成功（没有替换）：

 string.encode("iso8859-1", "utf-8",
     :fallback => {"\x80" => "123"}
 )
4

3 回答 3

1

这是我之前代码的一个非常快的版本，与 Ruby 1.8 和 1.9 兼容。

我可以用正则表达式识别无效的 utf8 字符，我只转换它们。

class String

  # Regexp for invalid UTF8 chars.
  # $1 will be valid utf8 sequence;
  # $3 will be the invalid utf8 char.
  INVALID_UTF8 = Regexp.new(
    '(([\xc0-\xdf][\x80-\xbf]{1}|' +
    '[\xe0-\xef][\x80-\xbf]{2}|' +
    '[\xf0-\xf7][\x80-\xbf]{3}|' +
    '[\xf8-\xfb][\x80-\xbf]{4}|' +
    '[\xfc-\xfd][\x80-\xbf]{5})*)' +
    '([\x80-\xff]?)', nil, 'n')

  if RUBY_VERSION >= '1.9'
    # ensure each char is utf8, assuming that
    # bad characters are in the +encoding+ encoding
    def utf8_ignore!(encoding)

      # avoid bad characters errors and encoding incompatibilities
      force_encoding('ascii-8bit')

      # encode only invalid utf8 chars within string
      gsub!(INVALID_UTF8) do |s|
        $1 + $3.force_encoding(encoding).encode('utf-8').force_encoding('ascii-8bit')
      end

      # final string is in utf-8
      force_encoding('utf-8')
    end

  else
    require 'iconv'

    # ensure each char is utf8, assuming that
    # bad characters are in the +encoding+ encoding
    def utf8_ignore!(encoding)

      # encode only invalid utf8 chars within string
      gsub!(INVALID_UTF8) do |s|
        $1 + Iconv.conv('utf-8', encoding, $3)
      end

    end
  end

end

# "\xe3" = "ã" in iso-8859-1
# mix valid with invalid utf8 chars, which is in iso-8859-1
a = "ãb\xe3"

a.utf8_ignore!('iso-8859-1')

puts a   #=> ãbã
于 2012-07-12T16:25:38.957 回答
1

我使用了以下代码（Ruby 1.8.7）。它测试每个 char >= 128 ASCII 以检查它是否是有效 utf-8 序列的开始。如果不是，则假定为 iso8859-1 并将其转换为 utf-8。

由于您的文件很大，这个过程可能会很慢！

class String
  # Grants each char in the final string is utf-8-compliant.
  # based on http://php.net/manual/en/function.utf8-encode.php#39986
  def utf8
    ret = ''

    # scan the string
    # I'd use self.each_byte do |b|, but I'll need to change i
    a = self.unpack('C*')
    i = 0
    l = a.length
    while i < l
      b = a[i]
      i += 1

      # if it's ascii, don't do anything.
      if b < 0x80
        ret += b.chr
        next
      end

      # check whether it's the beginning of a valid utf-8 sequence
      m = [0xc0, 0xe0, 0xf0, 0xf8, 0xfc, 0xfe]
      n = 0
      n += 1 until n > m.length || (b & m[n]) == m[n-1]

      # if not, convert it to utf-8
      if n > m.length
        ret += [b].pack('U')
        next
      end

      # if yes, check if the rest of the sequence is utf8, too
      r = [b]
      u = false

      # n bytes matching 10bbbbbb follow?
      n.times do
        if i < l
          r << a[i]
          u = (a[i] & 0xc0) == 0x80
          i += 1
        else
          u = false
        end
        break unless u
      end

      # if not, converts it!
      ret += r.pack(u ? 'C*' : 'U*')
    end

    ret
  end

  def utf8!
    replace utf8
  end
end

# let s be the string containing your file.
s2 = s.utf8

# or
s.utf8!
于 2012-07-09T13:09:36.410 回答
0

你在找这样的东西吗？

http://jalada.co.uk/2011/12/07/solving-latin1-and-utf8-errors-for-good-in-ruby.html

于 2012-07-09T13:04:15.647 回答