2

I've been using spirit classic for quite a while and recently start learning v2. Both are easy to use and powerful enough to handle moderate parsing issue.

In those idyllic days with the spirit classic and ascii characters, the job is simply adapting an EBNF to LL form. But finally, I have to deal with the i18n.

My thought is really crude. Make a forward iterator which iterates over the input text byte stream of any encoding. The iterator handles these encoding conversion job and feeds the parser with utf16/32 code unit(the iterator could be easily implemented by either iconv or icu4c).

The type of code unit should be the internal char type processed by parser. But after reading through the doc, I could find only several primitive, iterator templates have a explicit char_t parameter. Does that mean I have to reformulate those numerics, directives, scanners,and etc.?

I also checked out the v2 doc. It offers a namespace way to make everything consistent, but still not too much about how to explicitly change the internal char type. Again, I searched the mailing list archive, but seems those unicode and other encoding related post is still up in the air. Some one told me that spirit could still work through those files with different encoding. So I tested the parser using files with different encoding but same content. Several MBCS encoded files passed the test, and casually some utf8 file also passed. But other encoding failed most of the time.

4

1 回答 1

1

我怀疑您已经从 Boost Spirit 网站找到了 char_字符编码命名空间。

最后一页包含有用的评论

“我们为 Spirit 将支持的每个字符集都有一个命名空间。其中包括 ascii、iso8859_1、standard 和 standard_wide(以及未来的 unicode)。”

换句话说,目前 boost spirit 并不真正支持 unicode。它在他们的 TODO 清单上。

同时,您可以尝试像这样的笨拙的解决方法:

my_tag_ = lit("<") >> byte_ >> lit(">");

它将匹配恰好出现在尖括号之间的任何二进制字符串,前提是您不使用任何依赖于字符集的过滤器。

于 2011-12-19T17:02:48.383 回答