c++ - 使用 stringstream 标记具有不同分隔符的字符串

Question

如何使用 stringstream 来标记看起来像这样的行。

[标签] 操作码 [arg1] [,arg2]

标签可能并不总是存在，但如果不存在，则会有一个空白区域。操作码始终存在，并且操作码和 arg1 之间有一个空格或制表符。然后 arg1 和 arg2 之间没有空格，但它用逗号分隔。

此外，一些空白行上会有空白，因此需要丢弃它们。'#' 是注释

例如：

#Sample Input
TOP  NoP
     L   2,1
VAL  INT  0

这只是我将从中读取的文本文件的一个示例。因此，在第一行的标签中将是 TOP 并且操作码将 = NOP 并且没有传递任何参数。

我一直在努力，但我需要一种更简单的方法来标记化，从我所见，stringstream 似乎是我想使用的，所以如果有人能告诉我如何做到这一点，我我真的很感激。

我一直在绞尽脑汁思考如何做到这一点，只是为了向你展示我不只是不工作就问，这是我当前的代码：

int counter = 0;
int i = 0;
int j = 0;
int p = 0;

while (getline(myFile, line, '\n'))
{


    if (line[0] == '#')
    {
        continue;
    }

    if (line.length() == 0)
    {
        continue;
    }

    if (line.empty())
    {
        continue;
    }

    // If the first letter isn't a tab or space then it's a label

    if (line[0] != '\t' && line[0] != ' ')
    {

        string delimeters = "\t ";

        int current;
        int next = -1;


        current = next + 1;
        next = line.find_first_of( delimeters, current);
        label = line.substr( current, next - current );

        Symtablelab[i] = label;
        Symtablepos[i] = counter;

        if(next>0)
        {
            current = next + 1;
            next = line.find_first_of(delimeters, current);
            opcode = line.substr(current, next - current);


            if (opcode != "WORDS" && opcode != "INT")
            {
                counter += 3;
            }

            if (opcode == "INT")
            {
                counter++;
            }

            if (next > 0)
            {
                delimeters = ", \n\t";
                current = next + 1;
                next = line.find_first_of(delimeters, current);
                arg1 = line.substr(current, next-current);

                if (opcode == "WORDS")
                {
                    counter += atoi(arg1.c_str());
                }
            }

            if (next > 0)
            {
                delimeters ="\n";
                current = next +1;
                next = line.find_first_of(delimeters,current);
                arg2 = line.substr(current, next-current);

            }
        }

        i++;

    }

    // If the first character is a tab or space then there is no label and we just need to get a counter
    if (line[0] == '\t' || line[0] == ' ')
    {
        string delimeters = "\t \n";
        int current;
        int next = -1;
        current = next + 1;
        next = line.find_first_of( delimeters, current);
        label = line.substr( current, next - current );

    if(next>=0)
        {
            current = next + 1;
            next = line.find_first_of(delimeters, current);
            opcode = line.substr(current, next - current);

            if (opcode == "\t" || opcode =="\n"|| opcode ==" ")
            {
                continue;
            }

            if (opcode != "WORDS" && opcode != "INT")
            {
                counter += 3;
            }

            if (opcode == "INT")
            {
                counter++;
            }


            if (next > 0)
            {
                delimeters = ", \n\t";
                current = next + 1;
                next = line.find_first_of(delimeters, current);
                arg1 = line.substr(current, next-current);

                if (opcode == "WORDS")
                {
                    counter += atoi(arg1.c_str());
                }

            }



            if (next > 0)
            {
                delimeters ="\n\t ";
                current = next +1;
                next = line.find_first_of(delimeters,current);
                arg2 = line.substr(current, next-current);

            }
        }

    }
}

myFile.clear();
myFile.seekg(0, ios::beg);

while(getline(myFile, line))
{
    if (line.empty())
    {
        continue;
    }

    if (line[0] == '#')
    {
        continue;
    }

    if (line.length() == 0)
    {
        continue;
    }



    // If the first letter isn't a tab or space then it's a label

    if (line[0] != '\t' && line[0] != ' ')
    {

        string delimeters = "\t ";

        int current;
        int next = -1;


        current = next + 1;
        next = line.find_first_of( delimeters, current);
        label = line.substr( current, next - current );


        if(next>0)
        {
            current = next + 1;
            next = line.find_first_of(delimeters, current);
            opcode = line.substr(current, next - current);



            if (next > 0)
            {
                delimeters = ", \n\t";
                current = next + 1;
                next = line.find_first_of(delimeters, current);
                arg1 = line.substr(current, next-current);

            }

            if (next > 0)
            {
                delimeters ="\n\t ";
                current = next +1;
                next = line.find_first_of(delimeters,current);
                arg2 = line.substr(current, next-current);

            }
        }

        if (opcode == "INT")
        {
            memory[p] = arg1;
            p++;
            continue;
        }

        if (opcode == "HALT" || opcode == "NOP" || opcode == "P_REGS")
        {
            memory[p] = opcode;
            p+=3;
            continue;
        }

        if(opcode == "J" || opcode =="JEQR" || opcode == "JNE" || opcode == "JNER" || opcode == "JLT" || opcode == "JLTR" || opcode == "JGT" || opcode == "JGTR" || opcode == "JLE" || opcode == "JLER" || opcode == "JGE" || opcode == "JGER" || opcode == "JR")
        {
            memory[p] = opcode;
            memory[p+1] = arg1;
            p+=3;
            continue;
        }

        if (opcode == "WORDS")
        {
            int l = atoi(arg1.c_str());
            for (int k = 0; k <= l; k++)
            {
                memory[p+k] = "0";
            }

            p+=l;
            continue;
        }

        else
        {
            memory[p] = opcode;
            memory[p+1] = arg1;
            memory[p+2] = arg2;
            p+=3;
        }

    }

    // If the first character is a tab or space then there is no label and we just need to get a counter        


    if (line[0] == '\t' || line[0] == ' ')
    {
        string delimeters = "\t ";
        int current;
        int next = -1;
        current = next + 1;
        next = line.find_first_of( delimeters, current);
        label = line.substr( current, next - current );

    if(next>=0)
        {
            current = next + 1;
            next = line.find_first_of(delimeters, current);
            opcode = line.substr(current, next - current);

            if (opcode == "\t" || opcode =="\n"|| opcode ==" "|| opcode == "")
            {
                continue;
            }



            if (next > 0)
            {
                delimeters = ", \n\t";
                current = next + 1;
                next = line.find_first_of(delimeters, current);
                arg1 = line.substr(current, next-current);

            }



            if (next > 0)
            {
                delimeters ="\n\t ";
                current = next +1;
                next = line.find_first_of(delimeters,current);
                arg2 = line.substr(current, next-current);

            }
        }

        if (opcode == "INT")
        {
            memory[p] = arg1;
            p++;
            continue;
        }

        if (opcode == "HALT" || opcode == "NOP" || opcode == "P_REGS")
        {
            memory[p] = opcode;
            p+=3;
            continue;
        }

        if(opcode == "J" || opcode =="JEQR" || opcode == "JNE" || opcode == "JNER" || opcode == "JLT" || opcode == "JLTR" || opcode == "JGT" || opcode == "JGTR" || opcode == "JLE" || opcode == "JLER" || opcode == "JGE" || opcode == "JGER" || opcode == "JR")
        {
            memory[p] = opcode;
            memory[p+1] = arg1;
            p+=3;
            continue;
        }

        if (opcode == "WORDS")
        {
            int l = atoi(arg1.c_str());
            for (int k = 0; k <= l; k++)
            {
                memory[p+k] = "0";
            }

            p+=l;

            continue;
        }

        else
        {
            memory[p] = opcode;
            memory[p+1] = arg1;
            memory[p+2] = arg2;
            p+=3;
        }
    }
}

我显然想让这变得更好，所以任何帮助都将不胜感激。

Answer 1

在你为维护那些庞大的语句或尝试学习 Boost Spirit 而发疯之前if，让我们尝试编写一个非常简单的解析器。这篇文章有点长，没有直截了当，所以请多多包涵。

首先，我们需要一个语法，这似乎很简单：

    line
          label(optional)   opcode   argument-list(optional)

    argument-list
          argument
          argument, argument-list

英文：一行代码由一个可选标签、一个操作码和一个可选参数列表组成。参数列表可以是单个参数（整数），也可以是后跟分隔符（逗号）和另一个参数列表的参数。

让我们首先定义两个数据结构。标签应该是唯一的（对吗？），所以我们会有一组字符串，这样我们就可以随时轻松地查找它们，如果发现重复的标签可能会报告错误。下一个是字符串到 的映射size_t，它充当有效操作码的符号表以及每个操作码的预期参数数量。

std::set<std::string> labels;
std::map<std::string, size_t> symbol_table = {
    { "INT", 1},
    { "NOP", 0},
    { "L",   2}
};

我不知道memory您的代码到底是什么，但是您计算偏移量以计算将参数放在何处的方式似乎过于复杂。让我们定义一个可以优雅地保存一行代码的数据结构。我会做这样的事情：

typedef std::vector<int> arg_list;

struct code_line {
    code_line() : label(), opcode(), args() {}
    std::string  label;      // labels are optional, so an empty string
                             // will mean absence of label
    std::string  opcode;     // opcode, doh
    arg_list     args;       // variable number of arguments, it can be empty, too.
                             // It needs to match with opcode, we'll deal with
                             // that later
};

语法错误是一种不容易恢复的异常情况，所以让我们通过抛出异常来处理它们。我们简单的异常类可以如下所示：

struct syntax_error {
    syntax_error(std::string m) : msg(m) { }
    std::string msg;
};

标记化、词法分析和解析通常是独立的任务。但我想对于这个简单的例子，我们可以将分词器和词法分析器组合在一个类中。我们已经知道我们的语法是由哪些元素构成的，所以让我们编写一个类，它将输入作为文本并从中提取语法元素。界面可能如下所示：

class token_stream {
    std::istringstream stream; // stringstream for input
    std::string buffer;        // a buffer for a token, more on this later
public:
    token_stream(std::string str) : stream(str), buffer() { }

    // these methods are self-explanatory
    std::string get_label();
    std::string get_opcode();
    arg_list get_arglist();

    // we're taking a kind of top-down approach with this,
    // so let's forget about implementations for now
};

还有工作马，一个试图理解标记并code_line在一切正常时返回一个结构的函数：

code_line parse(std::string line)
{
    code_line temp;
    token_stream stream(line);

    // Again, self-explanatory, get a label, opcode and argument list from
    // token stream.

    temp.label = stream.get_label();
    temp.opcode = stream.get_opcode();
    temp.args = stream.get_arglist();

    // Everything went fine so far, remember we said we'd be throwing exceptions
    // in case of syntax errors.

    // Now we can check if we got the correct number of arguments for the given opcode:

    if (symbol_table[temp.opcode] != temp.args.size()) {
        throw syntax_error("Wrong number of parameters.");
    }

    // The last thing, if there's a label in the line, we insert it in the table.
    // We couldn't do that inside the get_label method, because at that time
    // we didn't yet know if the rest of the line is sintactically valid and a
    // exception thrown would have left us with a "dangling" label in the table.

    if (!temp.label.empty()) labels.insert(temp.label);

    return temp;
}

以下是我们如何使用这一切：

int main()
{
    std::string line;
    std::vector<code_line> code;

    while (std::getline(std::cin, line)) {

        // empty line or a comment, ignore it
        if (line.empty() || line[0] = '#') continue;

        try {
            code.push_back(parse(line));
        } catch (syntax_error& e) {
            std::cout << e.msg << '\n';

            // Give up, try again, log... up to you.
        }
    }
}

如果输入被成功解析，我们现在得到一个包含所有信息（标签、参数数量）的有效行向量，并且可以用它做几乎任何我们喜欢的事情。IMO，此代码将比您的代码更容易维护和扩展。例如，如果您需要引入新的操作码，只需在 map ( symbol_table) 中添加另一个条目。与你的if陈述相比如何？:)

唯一剩下的是token_streams 方法的实际实现。我是这样做的get_label：

std::string token_stream::get_label()
{
    std::string temp;

    // Unless the stream is empty (and it shouldn't be, we checked that in main),
    // operator>> for std::string is unlikely to fail. It doesn't hurt to be robust
    // with error checking, though

    if (!(stream >> temp)) throw ("Fatal error, empty line, bad stream?");

    // Ok, we got something. First we should check if the string consists of valid
    // characters - you probably don't want punctuation characters and such in a label.
    // I leave this part out for simplicity.

    // Since labels are optional, we need to check if the token is an opcode.
    // If that's the case, we return an empty (no) label.

    if (symbol_table.find(temp) != symbol_table.end()) {
        buffer = temp;
        return "";
    }

    // Note that above is where that `buffer` member of token_stream class got used.
    // If the token was an opcode, we needed to save it so get_opcode method can make
    // use of it. The other option would be to put the string back in the underlying 
    // stringstream, but that's more work and more code. This way, get_opcode needs   
    // to check if there's anything in buffer and use it, or otherwise extract from
    // the stringstream normally.

    // Check if the label was used before:

    if (labels.count(temp))
        throw syntax_error("Label already used.");

    return temp;
}

就是这样。我将其余的实现留给您作为练习。希望它有所帮助。:)

Answer 2

你肯定需要正则表达式，比如 boost regex；或用于此版本问题的词法分析和解析工具，例如 lex/yacc、flex/bison 或 boost spirit。

在这种复杂性下维护字符串和流是不值得的。