-1

为了创建一个自然语言计算器，我尝试了来自 nltk 的 TrigramTagger。我想在给定的句子中标记乘法和 2 个数字。例如：“什么是 5 和 7 的乘积”，这里的 'product' 是 'binary.multiply'，'5' 是 'num-1'，'7' 是 'num-2'。一旦我可以标记这 3 个，我就可以轻松计算答案。

但正如您在下面的输出中看到的那样，我无法针对这 2 个数字训练标注器。该数字必须与训练时完全相同，否则默认为“CD”。使用正则表达式或其他方式，我如何训练关于 num-1 和 num-2 的标记器？

import nltk.tag, nltk.data
from nltk import word_tokenize
default_tagger = nltk.data.load(nltk.tag._POS_TAGGER)

def evaluate(tagger, sentences):
    good,total = 0,0.
    for sentence in sentences:
        tags = tagger.tag(nltk.word_tokenize(sentence))
        print tags

train_sents = [
    [('product', 'binary.multiply'), ('of', 'IN'), ('5', 'num-1'), ('and', 'CC'), ('7', 'num-2'), ('?', '.')],
    [ ('what', 'WP'), ('is', 'VBZ'),  ('product', 'binary.multiply'), ('of', 'IN'), ('5', 'num-1'), ('and', 'CC'), ('7', 'num-2'), ('?', '.')],
    [('what', 'WP'), ('happens', 'NNS'), ('when', 'WRB'), ('I', 'PRP'), ('multiply', 'binary.multiply'), ('5', 'num-1'), ('with', 'IN'), ('7', 'num-2'), ('?', '.')],
    [('5', 'num-1'), ('*', 'binary.multiply'), ('3.2','CD')],
    [('is', 'NNP'), ('it', 'PRP'), ('possible', 'JJ'), ('to', 'TO'), ('multiply', 'binary.multiply'), ('5', 'num-1'), ('with', 'IN'), ('7', 'num-2'), ('?', '.')],
    [('what', 'WP'), ('is', 'VBZ'), ('5', 'num-1'), ('times', 'binary.multiply'), ('7', 'num-2'), ('?', '.')]
]

sentences = [
    ('product of 5 and 7?'),
    ('what is product of 3 and 2.7?'),
    ('what happens when I multiply 0.1 with 5.21?'),
    ('9.1 * 3.2'),
    ('is it possible to multiply 5 with 7?'),
    ('what is 5 times 7?')
]

tagger = nltk.TrigramTagger(train_sents, backoff=default_tagger)
evaluate(tagger, sentences)
#model = tagger._context_to_tag

这个程序的输出不能识别不同的数字为 num-1 和 num-2，如何让它识别呢？

[('product', 'binary.multiply'), ('of', 'IN'), ('4', 'CD'), ('and', 'CC'), ('2', 'CD'), ('?', '.')]
[('what', 'WP'), ('is', 'VBZ'), ('product', 'binary.multiply'), ('of', 'IN'), ('3', 'CD'), ('and', 'CC'), ('2.7', 'CD'), ('?', '.')]
[('what', 'WP'), ('happens', 'NNS'), ('when', 'WRB'), ('I', 'PRP'), ('multiply', 'binary.multiply'), ('0.1', 'CD'), ('with', 'IN'), ('5.21', 'CD'), ('?', '.')]
[('9.1', 'CD'), ('*', '-NONE-'), ('3.2', 'CD')]
[('is', 'NNP'), ('it', 'PRP'), ('possible', 'JJ'), ('to', 'TO'), ('multiply', 'binary.multiply'), ('2', 'CD'), ('with', 'IN'), ('77', 'CD'), ('?', '.')]
[('what', 'WP'), ('is', 'VBZ'), ('15', 'CD'), ('times', 'NNS'), ('72', 'CD'), ('?', '.')]
4

1 回答 1

1

嘿，我认为您可以为此使用RegexTagger，RegexTagger 是正则表达式标记器，根据匹配模式将标记分配给标记。

试试这个代码

from nltk.tokenize import word_tokenize
import nltk.tag
patterns = [
      (r'product|multiply|times', 'binary.multiply'),               # gerunds
      (r'\*', 'binary.multiply'), 
      (r'\d+\.\d+|\d+', 'num'),
      (r'\w+', 'word'),
      (r'[^0-9a-zA-z\*]','sym')
      ]
sentences = [
('product of 5 and 7?'),
('what is product of 3 and 2.7?'),
('what happens when I multiply 0.1 with 5.21?'),
('9.1 * 3.2'),
('is it possible to multiply 5 with 7?'),
('what is 5 times 7?')
]

regexp_tagger = nltk.RegexpTagger(patterns)
regexp_tagger.tag(word_tokenize(sentences[0]))

输出

[('product', 'binary.multiply'),
 ('of', 'NN'),
 ('5', 'num'),
 ('and', 'NN'),
 ('7', 'num'),
 ('?', 'sym')]

现在从输出列表中，您可以轻松找到第一个数字和第二个数字。

于 2016-01-18T12:13:43.647 回答