1

我有一个字符串，它由 Jericho HTML 解析器返回并包含一些俄语文本。根据source.getEncoding()相应 HTML 文件的标头，编码为 Windows-1251。

如何将此字符串转换为可读的内容？

我试过这个：

import java.io.UnsupportedEncodingException;

public class Program {
    public void run() throws UnsupportedEncodingException {
        final String windows1251String = getWindows1251String();
        System.out.println("String (Windows-1251): " + windows1251String);
        final String readableString = convertString(windows1251String);
        System.out.println("String (converted): " + readableString);
    }
    private String convertString(String windows1251String) throws UnsupportedEncodingException {
        return new String(windows1251String.getBytes(), "UTF-8");
    }
    private String getWindows1251String() {
        final byte[] bytes = new byte[] {32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, -17, -65, -67, -17, -65, -67, -17, -65, -67, -17, -65, -67, -17, -65, -67, -17, -65, -67, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32};
        return new String(bytes);
    }
    public static void main(final String[] args) throws UnsupportedEncodingException {
        final Program program = new Program();
        program.run();
    }
}

该变量bytes包含我的调试器中显示的数据，它是net.htmlparser.jericho.Element.getContent().toString().getBytes(). 我只是在此处复制并粘贴该数组。

这不起作用 -readableString包含垃圾。

如何修复它，即确保正确解码 Windows-1251 字符串？

更新 1（30.07.2015 12:45 MSK）：将调用中的编码更改convertString为 时Windows-1251，没有任何变化。请参阅下面的屏幕截图。

截屏

更新 2：另一次尝试：

第二张截图

更新 3（30.07.2015 14:38）：我需要解码的文本对应如下所示下拉列表中的文本。

预期结果

更新 4（30.07.2015 14:41）：编码检测器（代码见下文）说编码不是Windows-1251，而是UTF-8。

public static String guessEncoding(byte[] bytes) {
    String DEFAULT_ENCODING = "UTF-8";
    org.mozilla.universalchardet.UniversalDetector detector =
        new org.mozilla.universalchardet.UniversalDetector(null);
    detector.handleData(bytes, 0, bytes.length);
    detector.dataEnd();
    String encoding = detector.getDetectedCharset();
    System.out.println("Detected encoding: " + encoding);
    detector.reset();
    if (encoding == null) {
        encoding = DEFAULT_ENCODING;
    }
    return encoding;
}
4

3 回答 3

3

我通过修改从网站读取文本的代码来解决这个问题。

private String readContent(final String urlAsString) {
    final StringBuilder content = new StringBuilder();
    BufferedReader reader = null;
    InputStream inputStream = null;
    try {
        final URL url = new URL(urlAsString);
        inputStream = url.openStream();
        reader =
            new BufferedReader(new InputStreamReader(inputStream);

        String inputLine;
        while ((inputLine = reader.readLine()) != null) {
            content.append(inputLine);
        }
    } catch (final IOException exception) {
        exception.printStackTrace();
    } finally {
        IOUtils.closeQuietly(reader);
        IOUtils.closeQuietly(inputStream);
    }
    return content.toString();
}

我换了行

new BufferedReader(new InputStreamReader(inputStream);

至

new BufferedReader(new InputStreamReader(inputStream, "Windows-1251"));

然后它起作用了。

于 2015-07-30T12:33:03.287 回答
3

（根据更新，我删除了原来的答案并重新开始）

出现的文字

пїЅпїЅпїЅпїЅпїЅпїЅ

是对这些字节值的准确解码

-17, -65, -67, -17, -65, -67, -17, -65, -67, -17, -65, -67, -17, -65, -67, -17, -65, -67

（在两端用 32 填充，即空格。）

所以要么

1) 文字是垃圾或

2）文本应该看起来像那样或

3) 编码不是 Windows-1215

这条线明显错误

return new String(windows1251String.getBytes(), "UTF-8");

从字符串中提取字节并从中构造新字符串并不是在编码之间“转换”的一种方式。输入字符串和输出字符串都在内部使用 UTF-16 编码（通常您甚至不需要知道或关心这一点）。其他编码发挥作用的唯一时间是文本数据存储在字符串对象之外 - 即在您的初始字节数组中。转换发生在构造字符串然后完成时。没有从一种 String 类型到另一种的转换——它们都是一样的。

事实上，这

return new String(bytes);

和这个一样

return new String(bytes, "Windows-1251");

建议 Windows-1251 是平台默认编码。（您的时区是 MSK 进一步支持这一点）

于 2015-07-30T11:02:31.503 回答
1

只是为了确保您 100% 了解 java 如何处理char和byte.

byte[] input = new byte[1];

// values > 127 become negative when you put them in an array.
input[0] = (byte)239; // the array contains value -17 now.

// but all 255 values are preserved. 
// But if you cast them to integers, you should use their unsigned value.
// (casting alone isn't enough).
int output = input[0] & 0xFF; // output is 239 again

// you shouldn't cast directly from a single-byte to a char.
// because: char is 16-bit ; but you only want to use 1 byte ; unfortunately your negative values will be applied in the 2nd byte, and break it.
char corrupted = (char) input[0]; // char-code: 65519 (2 bytes are used)
char corrupted = (char) ((int)input[0]); // char-code: 66519 (2 bytes are used)

// just casting to an integer/character is ok for values < 0x7F though
// values < 0x7F are always positive, even when casted to byte
// AND the first 7-bits in any ascii-encodings (e.g. windows-1251) are identical.
byte simple = (byte) 'a';
char chr = (char) ascii_LT_7F; // will result in 'a' again

// But it's still more reliable to use the & 0xFF conversion.
// Because it ensures that your character can never be greater than char code 255 (a single byte), even when the byte is unexpectedly negative (> 0x7F).
char chr = (char) ((byte)simple & 0xFF); // also results in 'a'

// for value 239 (which is 0xEF) it's impossible though.
// a java char is 16-bit encoded internally, following the unicode character set.
// characters 0x00 to 0x7F are identical in most encodings.
// but e.g. 0xEF in windows-1251 does not match 0xEF in UTF-16.
// so, this is a bad idea.
char corrupted = (char) (input[0] & 0xFF);

// And that's something you can only fix by using encodings.
// It's good practice to use encodings really just ALWAYS.
// the encoding indicates what your bytes[] are encoded in NOW.
// your bytes will be converted to 16-bit characters.
String text = new String(bytes, "from-encoding");

// if you want to change that text back to bytes, use an encoding !!
// this time the encoding specifies is the TARGET-ENCODING.
byte[] bytes = text.getBytes("to-encoding");

我希望这有帮助。

至于显示的值： 我可以确认 byte[] 显示正确。我在 Windows-1251 代码页中检查了它们。（字节 -17 = int 239 = 0xEF = char 'п'）

换句话说，您的字节值不正确，或者它是不同的源编码。

于 2015-07-30T13:15:41.660 回答