我在大约 5000 个英语句子/段落上训练了一个 KENLM 语言模型。我想用两个或更多段查询这个 ARPA 模型，看看它们是否可以连接起来形成一个更长的句子，希望更“语法”。以下是我用来获取片段和“句子”的对数分数（以及基于 10 的幂值）的 Python 代码。我已经举了两个例子。显然，第一个例子中的句子比第二个例子中的句子更符合语法。但是，我的问题不是关于这个，而是关于如何将整个句子的语言模型分数与句子成分的分数联系起来。也就是说，如果句子在语法上优于其成分。
import math
import kenlm as kl
model = kl.LanguageModel(r'D:\seg.arpa.bin')
print ('************')
sentence = 'Mr . Yamada was elected Chairperson of'
print(sentence)
p1=model.score(sentence)
p2=math.pow(10,p1)
print(p1)
print(p2)
sentence = 'the Drafting Committee by acclamation .'
print(sentence)
p3=model.score(sentence)
p4=math.pow(10,p3)
print(p3)
print(p4)
sentence = 'Mr . Yamada was elected Chairperson of the Drafting Committee by acclamation .'
print(sentence)
p5=model.score(sentence)
p6=math.pow(10,p5)
print(p5)
print(p6)
print ('-------------')
sentence = 'Cases cited in the present volume ix'
print(sentence)
p1=model.score(sentence)
p2=math.pow(10,p1)
print(p1)
print(p2)
sentence = 'Multilateral instruments cited in the present volume xiii'
print(sentence)
p3=model.score(sentence)
p4=math.pow(10,p3)
print(p3)
print(p4)
sentence = 'Cases cited in the present volume ix Multilateral instruments cited in the present volume xiii'
print(sentence)
p5=model.score(sentence)
p6=math.pow(10,p5)
print(p5)
print(p6)
- ************ 先生 。Yamada was elected Chairperson of -34.0706558228 8.49853715087e-35 the Drafting Committee by acclamation . -28.3745193481 4.22163470933e-29 先生。Yamada was elected Chairperson of the Drafting Committee by acclamation . -55.5128440857 3.07012398337e-56 ------------- 本卷 ix 引用的案例 -27.7353248596 1.83939558773e-28 本卷 xiii 引用的多边文书 -34.4523620605 3.52888852435e-35本卷 ix 本卷 xiii 中引用的多边文书 -60.7075233459 1.9609957573e-61