我正在寻找一种将文本拆分为 n-gram 的方法。通常我会做类似的事情：
import nltk
from nltk import bigrams
string = "I really like python, it's pretty awesome."
string_bigrams = bigrams(string)
print string_bigrams
我知道 nltk 只提供二元组和三元组，但有没有办法将我的文本分成四克、五克甚至一百克？
谢谢！
问问题
216835 次
17 回答
257
其他用户给出的基于 Python 的优秀答案。但这是一种方法（以防万一，OP因重新发明图书馆nltk中已有的东西而受到惩罚）。nltk
有一个人们很少使用的ngram 模块nltk。这不是因为难以阅读 ngram，而是基于 n > 3 的 ngram 训练模型会导致大量数据稀疏。
from nltk import ngrams
sentence = 'this is a foo bar sentences and i want to ngramize it'
n = 6
sixgrams = ngrams(sentence.split(), n)
for grams in sixgrams:
print grams
于 2013-07-09T12:10:39.040 回答
76
我很惊讶这还没有出现：
In [34]: sentence = "I really like python, it's pretty awesome.".split()
In [35]: N = 4
In [36]: grams = [sentence[i:i+N] for i in xrange(len(sentence)-N+1)]
In [37]: for gram in grams: print gram
['I', 'really', 'like', 'python,']
['really', 'like', 'python,', "it's"]
['like', 'python,', "it's", 'pretty']
['python,', "it's", 'pretty', 'awesome.']
于 2013-07-08T16:54:06.313 回答
18
仅使用 nltk 工具
from nltk.tokenize import word_tokenize
from nltk.util import ngrams
def get_ngrams(text, n ):
n_grams = ngrams(word_tokenize(text), n)
return [ ' '.join(grams) for grams in n_grams]
示例输出
get_ngrams('This is the simplest text i could think of', 3 )
['This is the', 'is the simplest', 'the simplest text', 'simplest text i', 'text i could', 'i could think', 'could think of']
为了将 ngram 保持为数组格式，只需删除' '.join
于 2015-08-31T09:28:46.093 回答
18
人们已经很好地回答了你需要二元组或三元组的场景，但如果你需要每一个句子来表示这种情况，你可以使用nltk.util.everygrams
>>> from nltk.util import everygrams
>>> message = "who let the dogs out"
>>> msg_split = message.split()
>>> list(everygrams(msg_split))
[('who',), ('let',), ('the',), ('dogs',), ('out',), ('who', 'let'), ('let', 'the'), ('the', 'dogs'), ('dogs', 'out'), ('who', 'let', 'the'), ('let', 'the', 'dogs'), ('the', 'dogs', 'out'), ('who', 'let', 'the', 'dogs'), ('let', 'the', 'dogs', 'out'), ('who', 'let', 'the', 'dogs', 'out')]
如果你有一个限制，比如最大长度应该为 3 的三元组，那么你可以使用 max_len 参数来指定它。
>>> list(everygrams(msg_split, max_len=2))
[('who',), ('let',), ('the',), ('dogs',), ('out',), ('who', 'let'), ('let', 'the'), ('the', 'dogs'), ('dogs', 'out')]
您可以只修改 max_len 参数以实现任何克数，即四克、五克、六克甚至百克。
可以修改前面提到的解决方案以实现上述解决方案，但这个解决方案比这更直接。
进一步阅读请点击这里
当你只需要一个特定的克，比如二元组或三元组等时，你可以使用MAHassan的回答中提到的 nltk.util.ngrams 。
于 2018-12-18T10:39:59.387 回答
17
这是做 n-gram 的另一种简单方法
>>> from nltk.util import ngrams
>>> text = "I am aware that nltk only offers bigrams and trigrams, but is there a way to split my text in four-grams, five-grams or even hundred-grams"
>>> tokenize = nltk.word_tokenize(text)
>>> tokenize
['I', 'am', 'aware', 'that', 'nltk', 'only', 'offers', 'bigrams', 'and', 'trigrams', ',', 'but', 'is', 'there', 'a', 'way', 'to', 'split', 'my', 'text', 'in', 'four-grams', ',', 'five-grams', 'or', 'even', 'hundred-grams']
>>> bigrams = ngrams(tokenize,2)
>>> bigrams
[('I', 'am'), ('am', 'aware'), ('aware', 'that'), ('that', 'nltk'), ('nltk', 'only'), ('only', 'offers'), ('offers', 'bigrams'), ('bigrams', 'and'), ('and', 'trigrams'), ('trigrams', ','), (',', 'but'), ('but', 'is'), ('is', 'there'), ('there', 'a'), ('a', 'way'), ('way', 'to'), ('to', 'split'), ('split', 'my'), ('my', 'text'), ('text', 'in'), ('in', 'four-grams'), ('four-grams', ','), (',', 'five-grams'), ('five-grams', 'or'), ('or', 'even'), ('even', 'hundred-grams')]
>>> trigrams=ngrams(tokenize,3)
>>> trigrams
[('I', 'am', 'aware'), ('am', 'aware', 'that'), ('aware', 'that', 'nltk'), ('that', 'nltk', 'only'), ('nltk', 'only', 'offers'), ('only', 'offers', 'bigrams'), ('offers', 'bigrams', 'and'), ('bigrams', 'and', 'trigrams'), ('and', 'trigrams', ','), ('trigrams', ',', 'but'), (',', 'but', 'is'), ('but', 'is', 'there'), ('is', 'there', 'a'), ('there', 'a', 'way'), ('a', 'way', 'to'), ('way', 'to', 'split'), ('to', 'split', 'my'), ('split', 'my', 'text'), ('my', 'text', 'in'), ('text', 'in', 'four-grams'), ('in', 'four-grams', ','), ('four-grams', ',', 'five-grams'), (',', 'five-grams', 'or'), ('five-grams', 'or', 'even'), ('or', 'even', 'hundred-grams')]
>>> fourgrams=ngrams(tokenize,4)
>>> fourgrams
[('I', 'am', 'aware', 'that'), ('am', 'aware', 'that', 'nltk'), ('aware', 'that', 'nltk', 'only'), ('that', 'nltk', 'only', 'offers'), ('nltk', 'only', 'offers', 'bigrams'), ('only', 'offers', 'bigrams', 'and'), ('offers', 'bigrams', 'and', 'trigrams'), ('bigrams', 'and', 'trigrams', ','), ('and', 'trigrams', ',', 'but'), ('trigrams', ',', 'but', 'is'), (',', 'but', 'is', 'there'), ('but', 'is', 'there', 'a'), ('is', 'there', 'a', 'way'), ('there', 'a', 'way', 'to'), ('a', 'way', 'to', 'split'), ('way', 'to', 'split', 'my'), ('to', 'split', 'my', 'text'), ('split', 'my', 'text', 'in'), ('my', 'text', 'in', 'four-grams'), ('text', 'in', 'four-grams', ','), ('in', 'four-grams', ',', 'five-grams'), ('four-grams', ',', 'five-grams', 'or'), (',', 'five-grams', 'or', 'even'), ('five-grams', 'or', 'even', 'hundred-grams')]
于 2014-06-18T17:59:29.207 回答
8
您可以使用以下方法轻松创建自己的函数来执行此操作itertools：
from itertools import izip, islice, tee
s = 'spam and eggs'
N = 3
trigrams = izip(*(islice(seq, index, None) for index, seq in enumerate(tee(s, N))))
list(trigrams)
# [('s', 'p', 'a'), ('p', 'a', 'm'), ('a', 'm', ' '),
# ('m', ' ', 'a'), (' ', 'a', 'n'), ('a', 'n', 'd'),
# ('n', 'd', ' '), ('d', ' ', 'e'), (' ', 'e', 'g'),
# ('e', 'g', 'g'), ('g', 'g', 's')]
于 2013-07-08T16:50:30.110 回答
6
使用 python 的 builtin 构建二元组的更优雅的方法zip()。只需将原始字符串 by 转换为列表split()，然后正常传递列表一次，然后偏移一个元素。
string = "I really like python, it's pretty awesome."
def find_bigrams(s):
input_list = s.split(" ")
return zip(input_list, input_list[1:])
def find_ngrams(s, n):
input_list = s.split(" ")
return zip(*[input_list[i:] for i in range(n)])
find_bigrams(string)
[('I', 'really'), ('really', 'like'), ('like', 'python,'), ('python,', "it's"), ("it's", 'pretty'), ('pretty', 'awesome.')]
于 2016-03-29T05:22:16.300 回答
3
如果效率是一个问题，并且您必须构建多个不同的 n-gram（如您所说最多一百个），但您想使用纯 python，我会这样做：
from itertools import chain
def n_grams(seq, n=1):
"""Returns an itirator over the n-grams given a listTokens"""
shiftToken = lambda i: (el for j,el in enumerate(seq) if j>=i)
shiftedTokens = (shiftToken(i) for i in range(n))
tupleNGrams = zip(*shiftedTokens)
return tupleNGrams # if join in generator : (" ".join(i) for i in tupleNGrams)
def range_ngrams(listTokens, ngramRange=(1,2)):
"""Returns an itirator over all n-grams for n in range(ngramRange) given a listTokens."""
return chain(*(n_grams(listTokens, i) for i in range(*ngramRange)))
用法 ：
>>> input_list = input_list = 'test the ngrams generator'.split()
>>> list(range_ngrams(input_list, ngramRange=(1,3)))
[('test',), ('the',), ('ngrams',), ('generator',), ('test', 'the'), ('the', 'ngrams'), ('ngrams', 'generator'), ('test', 'the', 'ngrams'), ('the', 'ngrams', 'generator')]
~ 与 NLTK 相同的速度：
import nltk
%%timeit
input_list = 'test the ngrams interator vs nltk '*10**6
nltk.ngrams(input_list,n=5)
# 7.02 ms ± 79 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
%%timeit
input_list = 'test the ngrams interator vs nltk '*10**6
n_grams(input_list,n=5)
# 7.01 ms ± 103 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
%%timeit
input_list = 'test the ngrams interator vs nltk '*10**6
nltk.ngrams(input_list,n=1)
nltk.ngrams(input_list,n=2)
nltk.ngrams(input_list,n=3)
nltk.ngrams(input_list,n=4)
nltk.ngrams(input_list,n=5)
# 7.32 ms ± 241 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
%%timeit
input_list = 'test the ngrams interator vs nltk '*10**6
range_ngrams(input_list, ngramRange=(1,6))
# 7.13 ms ± 165 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
从我之前的回答转贴。
于 2018-01-18T07:52:05.737 回答
2
我从未处理过 nltk，但将 N-grams 作为一些小班项目的一部分。如果你想找到字符串中所有 N-gram 出现的频率，这里有一种方法可以做到这一点。D会给你你的N字的直方图。
D = dict()
string = 'whatever string...'
strparts = string.split()
for i in range(len(strparts)-N): # N-grams
try:
D[tuple(strparts[i:i+N])] += 1
except:
D[tuple(strparts[i:i+N])] = 1
于 2013-07-08T16:44:15.920 回答
2
对于four_grams，它已经在NLTK中，这里有一段代码可以帮助您实现这一点：
from nltk.collocations import *
import nltk
#You should tokenize your text
text = "I do not like green eggs and ham, I do not like them Sam I am!"
tokens = nltk.wordpunct_tokenize(text)
fourgrams=nltk.collocations.QuadgramCollocationFinder.from_words(tokens)
for fourgram, freq in fourgrams.ngram_fd.items():
print fourgram, freq
我希望它有所帮助。
于 2015-02-03T16:52:36.677 回答
2
您可以使用sklearn.feature_extraction.text.CountVectorizer：
import sklearn.feature_extraction.text # FYI http://scikit-learn.org/stable/install.html
ngram_size = 4
string = ["I really like python, it's pretty awesome."]
vect = sklearn.feature_extraction.text.CountVectorizer(ngram_range=(ngram_size,ngram_size))
vect.fit(string)
print('{1}-grams: {0}'.format(vect.get_feature_names(), ngram_size))
输出：
4-grams: [u'like python it pretty', u'python it pretty awesome', u'really like python it']
您可以设置为ngram_size任何正整数。即，您可以将文本拆分为四克、五克甚至一百克。
于 2015-08-14T19:57:39.537 回答
2
大约七年后，这里有一个更优雅的答案collections.deque：
def ngrams(words, n):
d = collections.deque(maxlen=n)
d.extend(words[:n])
words = words[n:]
for window, word in zip(itertools.cycle((d,)), words):
print(' '.join(window))
d.append(word)
print(' '.join(window))
words = ['I', 'am', 'become', 'death,', 'the', 'destroyer', 'of', 'worlds']
输出：
In [236]: ngrams(words, 2)
I am
am become
become death,
death, the
the destroyer
destroyer of
of worlds
In [237]: ngrams(words, 3)
I am become
am become death,
become death, the
death, the destroyer
the destroyer of
destroyer of worlds
In [238]: ngrams(words, 4)
I am become death,
am become death, the
become death, the destroyer
death, the destroyer of
the destroyer of worlds
In [239]: ngrams(words, 1)
I
am
become
death,
the
destroyer
of
worlds
于 2020-02-21T21:13:05.207 回答
2
如果您想要一个具有恒定内存使用的大字符串的纯迭代器解决方案：
from typing import Iterable
import itertools
def ngrams_iter(input: str, ngram_size: int, token_regex=r"[^\s]+") -> Iterable[str]:
input_iters = [
map(lambda m: m.group(0), re.finditer(token_regex, input))
for n in range(ngram_size)
]
# Skip first words
for n in range(1, ngram_size): list(map(next, input_iters[n:]))
output_iter = itertools.starmap(
lambda *args: " ".join(args),
zip(*input_iters)
)
return output_iter
测试：
input = "If you want a pure iterator solution for large strings with constant memory usage"
list(ngrams_iter(input, 5))
输出：
['If you want a pure',
'you want a pure iterator',
'want a pure iterator solution',
'a pure iterator solution for',
'pure iterator solution for large',
'iterator solution for large strings',
'solution for large strings with',
'for large strings with constant',
'large strings with constant memory',
'strings with constant memory usage']
于 2020-05-01T23:43:36.860 回答
1
Nltk 很棒，但有时对于某些项目来说是开销：
import re
def tokenize(text, ngrams=1):
text = re.sub(r'[\b\(\)\\\"\'\/\[\]\s+\,\.:\?;]', ' ', text)
text = re.sub(r'\s+', ' ', text)
tokens = text.split()
return [tuple(tokens[i:i+ngrams]) for i in xrange(len(tokens)-ngrams+1)]
示例使用：
>> text = "This is an example text"
>> tokenize(text, 2)
[('This', 'is'), ('is', 'an'), ('an', 'example'), ('example', 'text')]
>> tokenize(text, 3)
[('This', 'is', 'an'), ('is', 'an', 'example'), ('an', 'example', 'text')]
于 2015-08-28T14:48:28.303 回答
1
在 python 中做 n gram 很容易，例如：
def n_gram(list,n):
return [ list[i:i+n] for i in range(len(list)-n+1) ]
如果你这样做：
str = "I really like python, it's pretty awesome."
n_gram(str.split(" "),4)
你会得到
[['I', 'really', 'like', 'python,'],
['really', 'like', 'python,', "it's"],
['like', 'python,', "it's", 'pretty'],
['python,', "it's", 'pretty', 'awesome.']]
于 2021-11-01T05:49:44.413 回答
0
您可以使用以下代码获得所有 4-6 克，无需其他包：
from itertools import chain
def get_m_2_ngrams(input_list, min, max):
for s in chain(*[get_ngrams(input_list, k) for k in range(min, max+1)]):
yield ' '.join(s)
def get_ngrams(input_list, n):
return zip(*[input_list[i:] for i in range(n)])
if __name__ == '__main__':
input_list = ['I', 'am', 'aware', 'that', 'nltk', 'only', 'offers', 'bigrams', 'and', 'trigrams', ',', 'but', 'is', 'there', 'a', 'way', 'to', 'split', 'my', 'text', 'in', 'four-grams', ',', 'five-grams', 'or', 'even', 'hundred-grams']
for s in get_m_2_ngrams(input_list, 4, 6):
print(s)
输出如下：
I am aware that
am aware that nltk
aware that nltk only
that nltk only offers
nltk only offers bigrams
only offers bigrams and
offers bigrams and trigrams
bigrams and trigrams ,
and trigrams , but
trigrams , but is
, but is there
but is there a
is there a way
there a way to
a way to split
way to split my
to split my text
split my text in
my text in four-grams
text in four-grams ,
in four-grams , five-grams
four-grams , five-grams or
, five-grams or even
five-grams or even hundred-grams
I am aware that nltk
am aware that nltk only
aware that nltk only offers
that nltk only offers bigrams
nltk only offers bigrams and
only offers bigrams and trigrams
offers bigrams and trigrams ,
bigrams and trigrams , but
and trigrams , but is
trigrams , but is there
, but is there a
but is there a way
is there a way to
there a way to split
a way to split my
way to split my text
to split my text in
split my text in four-grams
my text in four-grams ,
text in four-grams , five-grams
in four-grams , five-grams or
four-grams , five-grams or even
, five-grams or even hundred-grams
I am aware that nltk only
am aware that nltk only offers
aware that nltk only offers bigrams
that nltk only offers bigrams and
nltk only offers bigrams and trigrams
only offers bigrams and trigrams ,
offers bigrams and trigrams , but
bigrams and trigrams , but is
and trigrams , but is there
trigrams , but is there a
, but is there a way
but is there a way to
is there a way to split
there a way to split my
a way to split my text
way to split my text in
to split my text in four-grams
split my text in four-grams ,
my text in four-grams , five-grams
text in four-grams , five-grams or
in four-grams , five-grams or even
four-grams , five-grams or even hundred-grams
你可以在这个博客上找到更多细节
于 2018-01-29T09:05:49.597 回答
0
这是一个老问题，但是如果您想将 n-gram 作为子字符串列表（而不是列表或元组列表）并且不想导入任何内容，则以下代码可以正常工作并且易于阅读：
def get_substrings(phrase, n):
phrase = phrase.split()
substrings = []
for i in range(len(phrase)):
if len(phrase[i:i+n]) == n:
substrings.append(' '.join(phrase[i:i+n]))
return substrings
您可以使用它，例如以这种方式获取术语列表的所有 n-gram，直到单词长度：
a = 5
terms = [
"An n-gram is a contiguous sequence of n items",
"An n-gram of size 1 is referred to as a unigram",
]
for term in terms:
for i in range(1, a+1):
print(f"{i}-grams: {get_substrings(term, i)}")
印刷：
1-grams: ['An', 'n-gram', 'is', 'a', 'contiguous', 'sequence', 'of', 'n', 'items']
2-grams: ['An n-gram', 'n-gram is', 'is a', 'a contiguous', 'contiguous sequence', 'sequence of', 'of n', 'n items']
3-grams: ['An n-gram is', 'n-gram is a', 'is a contiguous', 'a contiguous sequence', 'contiguous sequence of', 'sequence of n', 'of n items']
4-grams: ['An n-gram is a', 'n-gram is a contiguous', 'is a contiguous sequence', 'a contiguous sequence of', 'contiguous sequence of n', 'sequence of n items']
5-grams: ['An n-gram is a contiguous', 'n-gram is a contiguous sequence', 'is a contiguous sequence of', 'a contiguous sequence of n', 'contiguous sequence of n items']
1-grams: ['An', 'n-gram', 'of', 'size', '1', 'is', 'referred', 'to', 'as', 'a', 'unigram']
2-grams: ['An n-gram', 'n-gram of', 'of size', 'size 1', '1 is', 'is referred', 'referred to', 'to as', 'as a', 'a unigram']
3-grams: ['An n-gram of', 'n-gram of size', 'of size 1', 'size 1 is', '1 is referred', 'is referred to', 'referred to as', 'to as a', 'as a unigram']
4-grams: ['An n-gram of size', 'n-gram of size 1', 'of size 1 is', 'size 1 is referred', '1 is referred to', 'is referred to as', 'referred to as a', 'to as a unigram']
5-grams: ['An n-gram of size 1', 'n-gram of size 1 is', 'of size 1 is referred', 'size 1 is referred to', '1 is referred to as', 'is referred to as a', 'referred to as a unigram']
于 2021-12-28T15:00:28.210 回答