2

我正在使用nltk进行语言建模我将这篇文章用作mypet.txt文件中的语料库。对于大多数三元组，我得到 0.25 Kneser Ney 概率分布。我不知道为什么。这样对吗？为什么要这样做？这是我的word_ngram.py文件：

import io
import nltk
from nltk.util import ngrams
from nltk.tokenize import sent_tokenize
from preprocessor import utf8_to_ascii

with io.open("mypet.txt",'r',encoding='utf8') as utf_file:
    file_content = utf_file.read()

ascii_content = utf8_to_ascii(file_content)
sentence_tokenize_list = sent_tokenize(ascii_content)

all_tgrams = []
for sentence in sentence_tokenize_list:
    sentence = sentence.rstrip('.!?')
    tokens = nltk.re.findall(r"\w+(?:[-']\w+)*|'|[-.(]+|\S\w*", sentence)
    tgrams = ngrams(tokens, 3,pad_left=True,pad_right=True,left_pad_symbol='<s>', right_pad_symbol="</s>")
    all_tgrams.extend(tgrams)

frequency_distribution = nltk.FreqDist(all_tgrams)
kneser_ney = nltk.KneserNeyProbDist(frequency_distribution)
for i in kneser_ney.samples():
    print "{0}: {1}".format(kneser_ney.prob(i), i)

这是我处理 utf-8 字符的preprocessor.py文件：

# -*- coding: utf-8 -*-

import json


def utf8_to_ascii(utf8_text):
    with open("utf_to_ascii.json") as data_file:
        data = json.load(data_file)
    utf_table = data["chars"]
    for key, value in utf_table.items():
        utf8_text = utf8_text.replace(key, value)
    return utf8_text.encode('ascii')

这是我用来将 utf-8 char 替换为 ascii char 的utf_to_ascii.json文件：

{
 "chars": {
          "“":"",
          "”":"",
          "’":"'",
          "—":"-",
          "–":"-"
 }
}

这是几个三元组的示例输出：

0.25: ('side', '</s>', '</s>')
0.25: ('I', 'throw', 'a')
0.25: ('it', 'to', 'us')
0.25: ('guards', 'the', 'house')
0.0277777777778: ('<s>', 'I', 'am')
0.25: ('a', 'fire', 'broke')
0.125: ('our', 'house', 'at')
0.25: ('that', 'a', 'heap')
0.25: ('is', 'covered', 'with')
0.25: ('with', 'a', 'soft')
0.00862068965517: ('<s>', 'It', 'begins')
0.25: ('swim', '</s>', '</s>')
0.25: ('a', 'member', 'of')
0.25: ('bread', '</s>', '</s>')
0.25: ('love', '</s>', '</s>')
0.25: ('a', 'soft', 'fur')
0.25: ('body', 'is', 'covered')
0.25: ('I', 'bathe', 'it')
0.25: ('it', 'is', 'out')
0.25: ('<s>', 'A', 'thief')
0.25: ('go', 'hunting', '</s>')
0.025: ('It', 'is', 'loved')
0.25: ('it', 'a', 'loving')
0.25: ('with', 'soap', 'every-day')
0.25: ('other', 'members', 'of')
0.25: ('lying', 'there', 'was')
0.25: ('sensitive', 'to', 'sound')
0.25: ('and', 'the', 'flames')
0.25: ('kitchen', '</s>', '</s>')
0.25: ('strong', 'instinct', '</s>')
4

1 回答 1

2

可能到这个时候你已经想通了...无论如何我会使用nltk's搭配来代替。

您需要或多或少以相同的方式获取单词列表：

all_words = []
for sentence in sentence_tokenize_list:
    sentence = sentence.rstrip('.!?')
    tokens = nltk.re.findall(r"\w+(?:[-']\w+)*|'|[-.(]+|\S\w*", sentence)
    all_words.append('<s>')
    all_words.extend(tokens)
    all_words.append('</s>')
all_words[:9]

['<s>', 'I', 'am', 'fond', 'of', 'pets', '</s>', '<s>', 'I']

然后你创建你的搭配查找器：

from nltk.collocations import TrigramCollocationFinder, TrigramAssocMeasures
tcf = TrigramCollocationFinder.from_words(all_words)
trigram_measures = TrigramAssocMeasures()

以下将根据频率为您提供最好的 10 个三元组：

tcf.nbest(trigram_measures.raw_freq, 10)
[('</s>', '<s>', 'It'),
 ('<s>', 'It', 'is'),
 ('</s>', '<s>', 'I'),
 ('<s>', 'It', 'runs'),
 ('house', '</s>', '<s>'),
 ('</s>', '<s>', 'At'),
 ('It', 'is', 'very'),
 ('It', 'runs', 'after'),
 ('it', '</s>', '<s>'),
 ('the', 'house', '</s>')]

尽管您的数据集很小，但您通常希望过滤其中的一些，例如频率较低的。

tcf.apply_freq_filter(2)
tcf.apply_ngram_filter(lambda w1, w2, w3: ("<s>" in [w1, w2, w3]) and ("</s>" in [w1, w2, w3]))

tcf.nbest(trigram_measures.raw_freq, 10)

[('<s>', 'It', 'is'),
 ('<s>', 'It', 'runs'),
 ('It', 'is', 'very'),
 ('It', 'runs', 'after'),
 ('the', 'house', '</s>'),
 ('<s>', 'I', 'have'),
 ('<s>', 'It', 'has'),
 ('It', 'has', 'a'),
 ('member', 'of', 'the'),
 ('of', 'the', 'house')]

最后应用 Kneser Ney 平滑。

kneser_ney = nltk.KneserNeyProbDist(tcf.ngram_fd)
for i in tcf.nbest(trigram_measures.raw_freq, 10):
    print("{0}: {1}".format(kneser_ney.prob(i), i))

0.31896551724137934: ('<s>', 'It', 'is')
0.11206896551724138: ('<s>', 'It', 'runs')
0.225: ('It', 'is', 'very')
0.5625: ('It', 'runs', 'after')
0.5625: ('the', 'house', '</s>')
0.1388888888888889: ('<s>', 'I', 'have')
0.04310344827586207: ('<s>', 'It', 'has')
0.625: ('It', 'has', 'a')
0.625: ('member', 'of', 'the')
0.3125: ('of', 'the', 'house')

请注意，如果您只是使用三元组，这些是原始频率。

for i in tcf.nbest(trigram_measures.raw_freq, 10):
    print("{0}: {1}".format(tcf.score_ngram(trigram_measures.raw_freq, *i), i))

0.019083969465648856: ('<s>', 'It', 'is')
0.007633587786259542: ('<s>', 'It', 'runs')
0.0057251908396946565: ('It', 'is', 'very')
0.0057251908396946565: ('It', 'runs', 'after')
0.0057251908396946565: ('the', 'house', '</s>')
0.003816793893129771: ('<s>', 'I', 'have')
0.003816793893129771: ('<s>', 'It', 'has')
0.003816793893129771: ('It', 'has', 'a')
0.003816793893129771: ('member', 'of', 'the')
0.003816793893129771: ('of', 'the', 'house')
于 2017-04-11T16:12:39.940 回答