2

我想使用 gensim 从语料库中学习二元组，然后打印所学的二元组。我还没有看到这样做的例子。帮助表示赞赏

from gensim.models import Phrases
documents = ["the mayor of new york was there", "human computer interaction and machine learning has now become a trending research area","human computer interaction is interesting","human computer interaction is a pretty interesting subject", "human computer interaction is a great and new subject", "machine learning can be useful sometimes","new york mayor was present", "I love machine learning because it is a new subject area", "human computer interaction helps people to get user friendly applications"]
sentence_stream = [doc.split(" ") for doc in documents]

bigram = Phrases(sentence_stream)

# how can I print all bigrams learned and just the bigrams, including "new_york" and "human computer" ?enter code here
4

2 回答 2

3

如果您使用上述类训练模型Phrases并打印二元组而不保留模型，则 OP 的答案将起作用。当您保存模型然后在将来再次加载它时，它将不起作用。保存后加载模型时，您需要Phraser按如下方式使用该类：

from gensim.models.phrases import Phraser

然后加载模型：

bigram_model = Phraser.load('../../whatever_bigram_model')

然后，如果您确实使用以下方法作为提到的 OP 的答案，即

OP回答

import operator
sorted(
    {k:v for k,v in bigram_model.vocab.items() if b'_' in k if v>=bigram_model.min_count}.items(),
    key=operator.itemgetter(1),
    reverse=True)

您将收到一条错误消息：

AttributeError: 'Phraser' object has no attribute 'vocab'

解决方案

解决这个问题的方法是以下代码：

for bigram in bigram_model.phrasegrams.keys():
    print(bigram)

输出：

(b'word1', b'word2')
(b'word3', b'word4')

该解决方案适用于两种情况，对于持久和非持久模型，在 OP 给出的示例中，我的解决方案的修改版本是：

for ngrams, _ in bigram.vocab.items():
    unicode_ngrams = ngrams.decode('utf-8')
    if '_' in unicode_ngrams:
        print(unicode_ngrams)

给出：

the_mayor
mayor_of
of_new
new_york
york_was
was_there
human_computer
computer_interaction
interaction_and
and_machine
machine_learning
learning_has
has_now
now_become

输出中有更多内容，但我截断了它，以考虑这个答案的长度

我希望我的回答有助于增加清晰度。

于 2020-02-13T20:12:00.777 回答
0 
import operator
sorted(
    {k:v for k,v in bigram.vocab.items() if b'_' in k if v>=bigram.min_count}.items(),
    key=operator.itemgetter(1),
    reverse=True)
于 2018-12-09T16:44:09.793 回答