javascript - Is there a simple way I can tokenize a string without a full-blown lexer?

Question

I'm looking to implement the Shunting-yard Algorithm, but I need some help figuring out what the best way to split up a string into its tokens is.

If you notice, the first step of the algorithm is "read a token." This isn't exactly a non-trivial thing to do. Tokens can consist of numbers, operators and parens.

If you are doing something like:

(5+1)

A simple string.split() will give me an array of the tokens { "(", "5", "+", "1", ")" }.

However, it becomes more complicated if you have numbers with multiple digits such as:

((2048*124) + 42)

Now a naive string.split() won't do the trick. The multi-digit numbers are a problem.

I know I could write a lexer, but is there a way to do this without writing a full-blown lexer?

I'm implementing this in JavaScript and I'd like to avoid having to go down the lexer-path if possible. I'll be using the "*", "+", "-" and "/" operators, along with integers.

Answer 1

正则表达式呢？您可以轻松编写正则表达式以按照您想要的方式对其进行拆分，并且 JS string.split 方法也接受正则表达式作为参数。

例如...（修改以包含您需要的所有字符等）

/([0-9]+|[*+-\/()])/

Answer 2

您可以使用http://mikesamuel.blogspot.com/2009/05/efficient-parsing-in-javascript.html中所述的全局匹配

基本上，您创建一个描述令牌的正则表达式

/[0-9]+|false|true|\(|\)/g

并将'g'放在最后，使其全局匹配，然后调用它的匹配方法

var tokens = myRegex.match(inputString);

并取回一个数组。