首先,尝试使用 (i) 命名空间和 (ii) 明确的变量名称,例如:
>>> from nltk import PCFG
>>> from nltk.parse import ViterbiParser
>>> import urllib.request
>>> response = urllib.request.urlopen('https://raw.githubusercontent.com/salmanahmad/6.863/master/Labs/Assignment5/Code/wsjp.cfg')
>>> wsjp = response.read().decode('utf8')
>>> grammar = PCFG.fromstring(wsjp)
>>> parser = ViterbiParser(grammar)
>>> list(parser.parse('turn off the lights'.split()))
[ProbabilisticTree('S', [ProbabilisticTree('VP', [ProbabilisticTree('VB', ['turn']) (p=0.002082678), ProbabilisticTree('PRT', [ProbabilisticTree('RP', ['off']) (p=0.1089101771)]) (p=0.10768769667270556), ProbabilisticTree('NP', [ProbabilisticTree('DT', ['the']) (p=0.7396712852), ProbabilisticTree('NNS', ['lights']) (p=4.61672e-05)]) (p=4.4236397464693323e-07)]) (p=1.0999324002161311e-13)]) (p=2.5385077255727538e-14)]
如果我们看一下语法:
>>> grammar.check_coverage('please turn off the lights'.split())
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.4/dist-packages/nltk/grammar.py", line 631, in check_coverage
"input words: %r." % missing)
ValueError: Grammar does not cover some of the input words: "'please'".
要解决未知单词问题,有几种选择:
使用wildcard
非终端节点替换未知词。找到一些方法来用 替换语法没有涵盖的单词check_coverage()
,wildcard
然后用通配符解析句子
- 这通常会降低解析器的准确性,除非您使用处理未知单词的语法专门训练 PCFG,并且通配符是未知单词的超集。
返回到创建 PCFG 之前的语法生成文件,learn_pcfg.py
并在终端生成中添加所有可能的单词。
将未知词添加到您的 pcfg 语法中,然后重新规范化权重,给未知词赋予非常小的权重(您也可以尝试更智能的平滑/插值技术)
由于这是一个家庭作业问题,我不会给出完整代码的答案。但是上面的提示应该足以解决问题。