3

我正在尝试将 UTF-8 编码文件读入 UTF-32 (UCS-4) 字符串。基本上在内部，我希望应用程序内部有一个固定大小的字符。

在这里，我想确保翻译是作为流进程的一部分完成的（因为这是应该使用 Locale 的）。已经发布了替代问题来对字符串进行翻译（但这很浪费，因为您必须在内存中进行翻译阶段，然后您必须进行第二次传递才能将其发送到流中）。通过使用流中的语言环境进行操作，您只需执行一次传递，并且不需要制作副本（假设您想保留原始文件）。

这是我尝试过的。

#include <iostream>
#include <fstream>
#include <locale>
#include <codecvt>

int main()
{
    std::locale     converter(std::locale(), new std::codecvt_utf8<char32_t>);
    std::basic_ifstream<char32_t>   iFile;
    iFile.imbue(converter);
    iFile.open("test.data");

    std::u32string     line;
    while(std::getline(iFile, line))
    {
    }
}

由于这些都是标准类型，我对这个编译错误感到惊讶：

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/istream:275:41:
error: no matching function for call to 'use_facet'

            const ctype<_CharT>& __ct = use_facet<ctype<_CharT> >(__is.getloc());
                                        ^~~~~~~~~~~~~~~~~~~~~~~~~

编译：

g++ -std=c++14 test.cpp
4

1 回答 1

1

好像char32_t不是我想要的。只是搬到wchar_t为我工作。我怀疑这只适用于我想要的Linux系统和 Windows 这种转换将是 UTF-16 (UCS-2) （但我无法测试）。

int main()
{
   std::locale           utf8_to_utf32(std::locale(), new std::codecvt_utf8<wchar_t>);

    // Input stream reads UTF-8 and converts to UTF-32 (UCS-4) String
    std::wifstream        iFile("test.data");
    iFile.imbue(utf8_to_utf32);

    // Output UTF-32 (UCS-4) string converts to UTF-8 stream
    std::wofstream        oFile("test.res");
    oFile.imbue(utf8_to_utf32);


    // Now just read like you would normally.
    std::wstring     line;
    while(std::getline(iFile, line))
    {
        // UTF-32 characters are fixed size.
        // So reverse is simple just do it in-place.
        std::reverse(std::begin(line), std::end(line));

        // UTF-32 unfortunately also has grapheme clusters (these are groups of characters
        // that are displayed as a single glyph). By doing the reverse above we have split
        // these incorrectly. We need to do a second pass to reverse the characters inside
        // each cluster. This is beyond the scope of this question and left as an excursive
        // (but I may come back to it later).
        oFile << line << "\n";
    }
}

上面的评论表明这比读取数据比内联翻译要慢。所以我做了一些测试：

// read1.cpp 使用 codecvt 和 Locale 在流中翻译

#include <iostream>
#include <fstream>
#include <locale>
#include <codecvt>


int main()
{
    std::locale           utf8_to_utf32(std::locale(), new std::codecvt_utf8<wchar_t>);

    std::wifstream        iFile("test.data");
    iFile.imbue(utf8_to_utf32);

    std::wofstream        oFile("test.res1");
    oFile.imbue(utf8_to_utf32);

    std::wstring     line;
    while(std::getline(iFile, line))
    {
        std::reverse(std::begin(line), std::end(line));
        oFile << line << "\n";
    }
}

// read2.cpp 阅读后使用codecvt翻译。

#include <iostream>
#include <fstream>
#include <locale>
#include <codecvt>
#include <string>

int main()
{
    std::ifstream        iFile("test.data");
    std::ofstream        oFile("test.res2");

    std::wstring_convert<std::codecvt_utf8<wchar_t>> utf8_to_utf32;

    std::string     line;
    std::wstring    wideline;
    while(std::getline(iFile, line))
    {
        wideline = utf8_to_utf32.from_bytes(line);
        std::reverse(std::begin(wideline), std::end(wideline));
        oFile << utf8_to_utf32.to_bytes(wideline) << "\n";
    }
}

// read3.cpp 使用 UTF-8

#include <algorithm>
#include <iostream>
#include <string>
#include <fstream>

static bool is_lead(uint8_t ch) { return ch < 0x80 || ch >= 0xc0; }

/* Reverse a utf-8 string in-place */
void reverse_utf8(std::string& s) {
  std::reverse(s.begin(), s.end());
  for (auto p = s.begin(), end = s.end(); p != end; ) {
    auto q = p;
    p = std::find_if(p, end, is_lead);
    std::reverse(q, ++p);
  }
}

int main(int argc, char** argv)
{
    std::ifstream        iFile("test.data");
    std::ofstream        oFile("test.res3");

    std::string     line;
    while(std::getline(iFile, line))
    {
        reverse_utf8(line);
        oFile << line << "\n";
    }
    return 0;
}

测试文件是 58M 的 unicode Japanese

> ls -lah test.data
-rw-r--r--  1 loki  staff    58M Jan 28 11:28 test.data

> g++ -O3 -std=c++14 read1.cpp -o a1
> g++ -O3 -std=c++14 read2.cpp -o a2
> g++ -O3 -std=c++14 read3.cpp -o a3
>
> # This is the one using Locale in stream
> time ./a1

real    0m0.645s
user    0m0.521s
sys 0m0.108s
>
> # This is the one doing translation after reading.
> time ./a2

real    0m1.058s
user    0m0.916s
sys 0m0.123s
>
> # This is the one using UTF-8
> time ./a3

real    0m0.785s
user    0m0.663s
sys 0m0.104s

在流中进行翻译更快但并不明显（不是很多数据）。所以选择一个容易阅读的。

于 2016-01-27T20:08:18.207 回答