python - How to tokenize a string in C++ (like ply.lex in Python)?

Question

How do I easily tokenize a document or string using C++? I'm familiar with ply.lex module in Python but I couldn't find anything for C++. Is there a C++ library which can make use of lex and yacc? Or is there another better library or way of doing this easily?

Answer 1

flex和lex等价于ply.lex。示例flex：

%{
#include <stdio.h>
%}

%option noyywrap

%%
"+"               { puts("token: +"); }
"-"               { puts("token: -"); }
[0-9]+            { printf("token: %s\n", yytext); }
" "               { /* empty */ }
[\n|\r\n\t]       { /* empty */ }
.                 { fprintf(stderr, "Tokenizing error: synatx error '%c'\n", *yytext);
                    yyterminate(); }
%%

int main(int argc, char **argv)
{
  yylex();
  return 0;
}

编译：

> flex example.l
> gcc -Wall lex.yy.c -o lex
> lex
100 - 2 + 34
token: 100
token: -
token: 2
token: +
token: 34
>