3

我有一个 std::string 输出。使用 utf8proc 我想将其转换为有效的 utf8 字符串。 http://www.public-software-group.org/utf8proc-documentation

typedef int int32_t;
#define ssize_t int
ssize_t utf8proc_reencode(int32_t *buffer, ssize_t length, int options)
Reencodes the sequence of unicode characters given by the pointer buffer and length as UTF-8. The result is stored in the same memory area where the data is read. Following flags in the options field are regarded: (Documentation missing here) In case of success the length of the resulting UTF-8 string is returned, otherwise a negative error code is returned.
WARNING: The amount of free space being pointed to by buffer, has to exceed the amount of the input data by one byte, and the entries of the array pointed to by str have to be in the range of 0x0000 to 0x10FFFF, otherwise the program might crash!

那么首先，我如何在最后添加一个额外的字节？那么如何从 std::string 转换为 int32_t *buffer？

这不起作用：

std::string g = output();
fprintf(stdout,"str: %s\n",g.c_str());
g += " ";   //add an extra byte?? 
g = utf8proc_reencode((int*)g.c_str(), g.size()-1, 0);
fprintf(stdout,"strutf8: %s\n",g.c_str());
4

1 回答 1

0

您很可能实际上并不想要utf8proc_reencode()-该函数采用有效的 UTF-32 缓冲区并将其转换为有效的 UTF-8 缓冲区，但是由于您说您不知道数据的编码是什么，所以您不能使用那个功能。

所以，首先你需要弄清楚你的数据实际上是什么编码。你可以使用http://utfcpp.sourceforge.net/来测试你是否已经有有效的 UTF-8 和utf8::is_valid(g.begin(), g.end()). 如果这是真的，你就完了！

如果为假，事情就会变得复杂......但 ICU ( http://icu-project.org/ ) 可以帮助你；见http://userguide.icu-project.org/conversion/detection

一旦您稍微可靠地知道您的数据采用什么编码，ICU 可以再次帮助将其转换为 UTF-8。例如，假设您的源数据g采用 ISO-8859-1： 

UErrorCode err = U_ZERO_ERROR; // check this after every call...
// CONVERT FROM ISO-8859-1 TO UChar
UConverter *conv_from = ucnv_open("ISO-8859-1", &err);
std::vector<UChar> converted(g.size()*2); // *2 is usually more than enough
int32_t conv_len = ucnv_toUChars(conv_from, &converted[0], converted.size(), g.c_str(), g.size(), &err);
converted.resize(conv_len);
ucnv_close(conv_from);
// CONVERT FROM UChar TO UTF-8
g.resize(converted.size()*4);
UConverter *conv_u8 = ucnv_open("UTF-8", &err);
int32_t u8_len = ucnv_fromUChars(conv_u8, &g[0], g.size(), &converted[0], converted.size(), &err);
g.resize(u8_len);
ucnv_close(conv_u8);
之后，您g现在持有 UTF-8 数据。

于 2013-01-06T16:51:40.963 回答